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Publication Notes 


The 2018 NAEP Technology and Engineering Literacy Framework is the same frame- 
work first developed for the 2014 NAEP Technology and Engineering Literacy Assess- 
ment. Continuity in the NAEP Technology and Engineering Literacy Framework enables 
reporting of student achievement trends over time. This edition reflects updated dates 
and references to legislation, National Assessment Governing Board actions, and NAEP 
activities. 
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PREFACE 


We live in a world that is, to a large extent, shaped by technology. Our computers and our smart- 
phones: technology. Our automobiles and airplanes: technology. Our homes and offices, our food 
and clothing, our heating and cooling, our entertainment, our medical care: They are all created and 


driven by technology. 


Likewise, many of the critical challenges that we face as a society, and that our young people will 
eventually need to address, have large technological components such as the quest to link experts 
throughout the world, the search for sustainable energy, dealing with global pandemics, and the 


development of environmentally friendly agriculture to feed a growing world population. 


Despite its importance, however, technology has not been a focus of instruction and assessment 
in our educational system, particularly at the elementary and secondary levels. Some technologies 
are being integrated into humanities, social science, science, and mathematics classes as methods 
for supporting learning in these subjects. Technologies are also becoming subjects of study, with 
the goal of developing an understanding of the technologies themselves, their various roles, and the 
engineering design processes to address human needs and wants. However, there are no standard- 
ized, national assessments to provide evidence of what technologically literate K-12 students know 
about technology and engineering and the roles they play in our lives, or the extent to which stu- 


dents can use technologies and understand how engineers design and develop them. 


Technology and engineering are increasingly being incorporated into school coursework, rang- 
ing from instruction on the use of computers and information technology within school subjects 
to classes that examine the role of technology in society, or courses that teach engineering design. 
Information communication technologies have become integral tools of the trade in academic, 
workplace, and practical contexts. Technology and engineering are essential components of contem- 


porary science, technology, engineering, and mathematics (STEM) education. 


Because of this growing importance of technology and engineering in the educational landscape, 
the National Assessment Governing Board decided that an assessment of technological literacy 
would be an important addition to the National Assessment of Educational Progress (NAEP). For 
more than 35 years, NAEP has assessed achievement in a variety of key subjects by testing samples 


of students most often in the 4th, 8th, and 12th grades. The results, commonly referred to as The 
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Nation’s Report Card, have become an important source of information on what US. students know 


and are able to do in a range of subject areas. 


The Governing Board sought a framework of technological literacy knowledge and skills that 
identifies the understandings and applications of technology principles that are important for all 
students. The framework focuses on “literacy” as the level of knowledge and competencies needed 
by all students and citizens. People who are literate about technology and engineering are not 
expected to “do” engineering or produce technology in the professional sense. Therefore, the frame- 
work is not intended to address technical knowledge of specific technologies or types of engineer- 
ing expertise taught in specialized courses to prepare some students for postsecondary engineering 
studies. At grade 4, for example, all students can be expected to identify types of technologies in 
their world, design and test a simple model, explain how technologies can result in positive and 
negative effects, and use common technologies to achieve goals in school and their everyday life. By 
grade 12, for example, all students can be expected to select and use a variety of tools and media to 
conduct research, evaluate how well a solution meets specified criteria, and develop a plan to address 


a complex global issue. 


The Governing Board chose the contractor WestEd to recommend a framework for this new 
assessment. WestEd in turn assembled a broad array of individuals and organizations to take part in 
the effort. 


The resulting framework is the culmination of a long, complex process that drew on the contri- 
butions of thousands of individuals and organizations with expertise in the use of technology for 
learning, technology education, and engineering. Eighteen outreach meetings held over a 15-month 
period gathered feedback from professional organizations, practitioners, and the general public. 


Surveys documented comments on drafts of the framework. 


As the framework was being developed, it became clear that the terms “technology,” “engineer- 
ing,’ “information communication technology, “21st-century skills” and “literacy” are defined and 
used in significantly different ways in formal and informal education, in standards, by professional 
organizations, and in legislation. Therefore, the framework development committees recommended 
a change of the framework title from “technological literacy” to “technology and engineering 
literacy” to encompass general literacy about the use, effects, and designing of technologies. The 
2018 NAEP Technology and Engineering Literacy Framework is a statement about what should be 
expected of students in terms of their knowledge and skills with technology, written to be the basis 
for an assessment of technology and engineering literacy appropriate for all students. It opens the 
door to seeing what our K-12 students know about technology and engineering, in the same way 
that NAEP assesses their knowledge and capabilities in reading, mathematics, science, and other 


subjects. 
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The 2018 NAEP Technology and Engineering Literacy Assessment will use the same framework 
used in 2014. The 2014 NAEP Technology and Engineering Literacy Assessment started a new 
NAEP trend (ie., measure of student progress in technology and engineering literacy), and the 2018 
NAEP Technology and Engineering Literacy Report Card will include student performance trends 
from 2014 to 2018. 


This framework describes the assessment of technology and engineering literacy at grades 4, 8, 
and 12, although not all three grades will be included in the initial assessment. The 2014 NAEP 


Technology and Engineering Literacy Assessment was conducted at grade 8. 


Technology and Engineering Literacy 


Any assessment of students’ technology and engineering literacy must start with a clear idea of 
exactly what technology and engineering literacy means. That in turn requires clear definitions of 


technology and engineering. 


Research shows that most Americans associate technology with computers and related elec- 
tronic devices. However, while the computer is certainly an important example of technology—and 
one that plays an especially important role in this framework—historically the term “technology” 
has had a much broader and deeper meaning, and it is this meaning that is represented in this 


framework: 


Technology is any modification of the natural world done to fulfill human needs or 


desires. 


This definition sees technology as encompassing the entire human-made world, from the 
simplest artifacts, such as paper and pencil, to the most complex—buildings and cities, the electric 
power grid, satellites, and the Internet. Furthermore, technology is not just the things that people 
create. It includes the entire infrastructure needed to design, manufacture, operate, and repair tech- 
nological artifacts, from corporate headquarters and engineering schools to manufacturing plants, 


media outlets, and distribution networks. 


Engineers are the agents for designing the many technologies that modify the world. Engineers 
may not actually construct artifacts such as calculators, bridges, or airplanes. Engineers develop 
the plans and directions for how artifacts are to be constructed, as well as design processes such as 


assembly lines or procedures for clinical trials of pharmaceuticals. 
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The framework uses the following definition of engineering: 


Engineering is a systematic and often iterative approach to designing objects, processes, 


and systems to meet human needs and wants. 
This framework defines technology and engineering literacy in a broad fashion: 


Technology and engineering literacy is the capacity to use, understand, and evaluate 
technology as well as to understand technological principles and strategies needed to 


develop solutions and achieve goals. 


Thus, technology and engineering literacy has much in common with scientific, mathematical, 
and language literacy. As with these other forms of academic literacy, technology and engineering 
literacy involves the mastery of a set of tools needed to participate intelligently and thoughtfully in 
society. The tools are different, but the ultimate goal is the same. One particularly important set of 
technological tools consists of information and communication technologies, and this framework 
sees knowledge of these particular technologies as an integral part of technology and engineering 


literacy. 


Content 


This framework describes the essential knowledge and skills that will be assessed on the NAEP 
Technology and Engineering Literacy Assessment, beginning in 2014. It is not possible to assess 
every facet of technology and engineering literacy—the field is far too broad—so the assessment 
targets have been selected to capture those aspects of the nature, processes, uses, and effects of tech- 
nology that are particularly important to participation in the economic, civic, and social spheres of 


modern society. 
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Areas of Technology and Engineering Literacy 


The assessment targets are organized into three 
major areas of technology and engineering literacy, each 
corresponding to a broad body of knowledge and skills 
with which students should be familiar: 


e Technology and Society 
e Design and Systems 


e Information and Communication Technology 


Technology and Society deals with the effects that 
technology has on society and the natural world and 
with the sorts of ethical questions that arise from those 


effects. Knowledge and capabilities in this area are 


crucial for understanding the issues surrounding the 


development and use of various technologies and for participating in decisions regarding their use. 


Design and Systems covers the nature of technology, the engineering design process by which 
technologies are developed, and basic principles of dealing with everyday technologies, including 
maintenance and troubleshooting. An understanding of the design process is particularly valuable 


in assessing technologies, and it can also be applied in areas outside technology, since design is a 


broadly applicable skill. 


Information and Communication Technology includes computers and software learning tools, 
networking systems and protocols, hand-held digital devices, and other technologies for accessing, 
creating, and communicating information and for facilitating creative expression. Although it is just 
one among several types of technologies, it has achieved a special prominence in technology and 
engineering literacy because familiarity and facility with it is essential in virtually every profession in 


modern society. 


These three areas of technology and engineering literacy are closely interconnected. For exam- 
ple, to address an issue related to technology and society such as clean water, energy needs, dig- 
ital-age communications, or global climate change, it is important to have an understanding of 
technological systems and the design process. Similarly, one must also be able to use various infor- 
mation and communication technologies to research problems, develop and communicate possible 


solutions, and achieve goals. 
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Practices 


In all three areas of technology and engineering literacy, students are expected to be able to 
apply particular ways of thinking and reasoning when approaching a problem. These types of think- 
ing and reasoning are referred to as practices, and the framework specifies three kinds in particular 


that students are expected to demonstrate when responding to test questions: 


e Understanding Technological Principles 
e Developing Solutions and Achieving Goals 


e Communicating and Collaborating 


Understanding Technological Principles focuses on how well students are able to make use of 
their knowledge about technology. This knowledge ranges from discrete declarative facts and con- 
cepts to higher-level reasoning about how facts, concepts, and principles are organized into various 
structures and relationships. Students should be able to use their knowledge about technology to 


develop explanations and make predictions, comparisons, and evaluations. 


Developing Solutions and Achieving Goals refers to students’ systematic use of technological 
knowledge, tools, and skills to solve problems and achieve goals presented in realistic contexts. 
This practice includes both procedural and strategic capabilities—knowing how to apply simple 
steps and use technological tools to address real situations, as well as when and where to apply the 


tools and processes. 


Communicating and Collaborating concerns how well students are able to use contemporary 
technologies to communicate for a variety of purposes and in a variety of ways, working individually 


or in teams, with peers and experts. 


Contexts 


Technology and engineering literacy requires not just that students know about technology 
but also that they are able to recognize the technologies around them and apply what they know 
to problems and projects involving particular technologies. Consequently, NAEP Technology and 
Engineering Literacy Assessment items will measure students’ technology and engineering literacy 
in the contexts of relevant societal issues and of actual problems that people are commonly called on 


to solve and goals they seek to achieve. 


The contexts will vary according to the area of technology and engineering literacy and the prac- 
tice being assessed, but they should include a broad sampling from the major technological fields in 
use today: medical technologies, information and communication technologies, energy and power 


technologies, transportation technologies, agriculture and biotechnology, and so forth. 
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Types of Tasks and Items 


Technology and engineering literacy involves a range of knowledge and capabilities whose 
assessment requires having students perform a variety of actions using diverse tools to solve prob- 
lems and meet goals within rich, complex scenarios that reflect realistic situations. To learn what 
students know and can do with regard to technology and engineering, the framework calls for the 
assessment to be totally computer-based. The use of computers makes it feasible to utilize scenar- 
io-based multimedia tasks developed for the assessment, along with more traditional item types. 
Although many items will be selected-response items, such as multiple choice, in which only the 
final answer is observed, there will also be a number of more complex items that allow the moni- 
toring of the students’ actions as they manipulate components of the systems and models that are 


presented to them. 


Reporting Results 


Results of the NAEP Technology and Engineering Literacy Assessment will be reported in 
terms of percentages of students who attain each of the three achievement levels— Basic, Proficient, 
and Advanced—as is typical for NAEP results in other subjects. In addition, results of the assess- 
ment will be reported in terms of NAEP scale scores, on a 0-300 scale, for example. Scale scores will 
be reported for each of the three major assessment areas of technology and engineering literacy: 
Technology and Society, Design and Systems, and Information and Communication Technology. 
Scale scores will be reported by the three practices addressed in the Technology and Engineering 
Literacy Assessment: Understanding Technological Principles; Developing Solutions and Achieving 


Goals; and Communicating and Collaborating. An overall composite score will also be reported. 


In addition to scores, data will be collected on student gender, race/ethnicity, eligibility for 
free and reduced-price lunch, disabilities, and English language learner status, as well as a series of 
contextual variables specific to this content area. This enables analysis of the achievement data based 
on these factors. As the assessment is administered in years beyond the initial assessment in 2014, a 


trend line can be established to track changes in student achievement over time. 


Development of the Framework 


The development of the NAEP Technology and Engineering Literacy Framework required the 
participation of a broad array of individuals and organizations to consider the various definitions 
of “technology,” “engineering,” and “technology and engineering literacy,” and to devise a single 
framework that takes all of them into account. Among the organizations whose representatives 
provided advice and feedback were the Council of Chief State School Officers, the International 
Society for Technology in Education, the International Technology and Engineering Educators 
Association, the National Academy of Engineering, the Partnership for 21st Century Skills, and the 
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State Educational Technology Directors Association. Eighteen outreach meetings were held across 
the country to solicit feedback on drafts of the framework, with approximately 2,000 participants. 
Online and paper-based surveys provided specific written feedback from 350 respondents. 


In determining what students should know and be able to do in the areas of technology and 
engineering and to set forth criteria for the design of the assessment, the framework developers drew 
from a wide variety of sources. These included state, national, and international technology and 
engineering standards; research studies and reports; and a broad array of publications on technology 
and engineering literacy, educational standards, and assessment. To allow for some in-depth probing 
of fundamental knowledge and skills, the framework and the specifications represent a distillation 
rather than a complete representation of the universe of achievement outcomes targeted by teaching 


about technology and engineering. 


Developers of the framework discussed various constraints regarding exactly what can be 
measured. In particular, there are important aspects of technology and engineering literacy that are 
difficult and time-consuming to measure—such as habits of mind, sustained design and research 
projects, and working as part of a team—but that are valued by engineers, technology and engineer- 
ing educators, and the business community. These can be only partially represented in the NAEP 
Technology and Engineering Literacy Assessment. 


Developing a technology and engineering literacy assessment from this framework will pose a 
number of challenges. For example, because of the current underdeveloped state of assessing tech- 
nology and engineering literacy, there are few existing sample tasks to serve as examples for assess- 


ment development. 


The framework developers believe this assessment will provide a rich and accurate measure of 
the technology and engineering literacy that students need both for their schooling and for their 
lives. Development of these technology and engineering literacy skills and capabilities is the respon- 
sibility of all teachers—not only technology and engineering educators but also teachers across the 


curriculum—and also involves the expectations of parents and society. 
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CHAPTER ONE 


OVERVIEW 


Need for a Framework in Technology and Engineering Literacy 


To what extent can young people analyze the pros and cons of a proposal to develop a new 
source of energy? Construct and test a model or prototype? Use the Internet to find and summarize 


data and information to solve a problem or achieve a goal? 


The exploding growth in the world of technology and the need to answer questions similar to 
those above led the National Assessment Governing Board to sponsor the development of a frame- 
work for a national assessment of technology and engineering literacy. For generations students have 
been taught about technology and have been instructed in the use of various technological devices, 
but there has been no way to know exactly what students understand about technologies and their 


effective use. 


As the framework was being developed, it became clear that the terms “technology,” “engineer- 
ing,” “information communication technology,’ “21st-century skills,” and “literacy” are defined and 
used in significantly different ways in formal and informal education, in standards, by professional 
organizations, and in legislation. Therefore, the framework development committees recommended 
a change of the framework title from “technological literacy” to “technology and engineering 
literacy” to encompass general literacy about the use, effects, and designing of technologies. The 
2018 NAEP Technology and Engineering Literacy Framework is a statement about what should be 
expected of students in terms of their knowledge and skills with technology, written to be the basis 
for an assessment of technology and engineering literacy appropriate for all students. It opens the 
door to seeing what our K-12 students know about technology and engineering, in the same way 
that NAEP assesses their knowledge and capabilities in reading, mathematics, science, and other 


subjects. 
Technology and engineering literacy has been defined in a variety of ways, but here it will be 


thought of as the capacity to use, manage, and evaluate the technologies that are most relevant in 


one’s life, including the information and communication technologies that are particularly salient in 
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the world today, as well as to understand technological principles and strategies needed to develop 


solutions and achieve goals. 


Because technology is such a crucial component of modern society, it is important that students 
develop an understanding of its range of features and applications, the design process that engineers 
use to develop new technological devices, the trade-offs that must be balanced in making decisions 
about the use of technology, and the way that technology shapes society and society shapes technol- 
ogy. Indeed, some have argued that it is time for technology and engineering literacy to take its place 
alongside the traditional literacies in reading, mathematics, and science as a set of knowledge and 
skills that students are expected to develop during their years in school. Others go further in concep- 
tualizing “new literacies” in which technologies are seen to restructure and support the development 
of academic and workplace expertise through “cyberlearning” in the networked world (National 
Science Foundation [NSF], 2008). 


As of 2008, all 50 states were required to report to the US. Department of Education on 
technology literacy, using Information and Communication Technology (ICT) standards based 
on the National Education Technology Standards for Students (NETSYS) (International Society 
for Technology in Education [ISTE], 2007). Seven states have formal assessments for technology 
literacy (Metiri Group, 2009). The Standards for Technological Literacy developed by the Interna- 
tional Technology and Engineering Educators Association (ITEEA) were being used in 41 states 
for technology education courses at either the state level or in local school districts (Dugger, 2007). 
A dozen states required technology education for students in at least some grades; 22 states offered 
technology education as an elective. Engineering-based technology education may be offered as a 


separate subject or embedded into other subject areas, such as science or social studies. 


Reflecting the increasing importance of the role of technologies in 21st-century life, the 
National Assessment Governing Board decided to develop a framework for a national assessment 
of technology and engineering literacy. In announcing the NAEP Technology and Engineering 
Literacy Assessment, the Governing Board stated that the goals and objectives of the framework 
should be based on the future needs of the nation and individuals, and on the levels of technological 


literacy likely to be expected of students in the first half of the 21st century. 


The primary audience for this framework is the general public. For that reason, the develop- 
ers have attempted to use a minimal amount of technical language and have provided definitions 
when employing terms with which readers may not be familiar. There is also a glossary of terms in 
appendix A. The framework contains sufficient information for policymakers, educators, and other 
interested parties to understand the nature and scope of the assessment. Those interested in more 
technical information can consult the companion document, Assessment and Item Specifications for 


the NAEP Technology and Engineering Literacy Assessment. 


It is important to note that this framework is an assessment framework, not a curriculum frame- 
work. That is, although it provides a definition of what students should know and be able to do 
concerning technology, and how NAEP will measure that content, it is not intended to tell teachers 
and administrators what should be taught in the classroom, when it should be taught, or how it 


should be taught. 


In addition to laying out the need for a technology and engineering literacy assessment frame- 
work, this chapter offers background information on NAEP and provides a list of resources used to 
develop the framework, definitions of key terms, a description of the process used for developing the 
framework, an explanation of the challenges of developing a technology and engineering literacy 


assessment framework, and an overview of the framework. 


Background of NAEP 


For more than 35 years, NAEP has measured student achievement nationally, state by state, 
and among 20 large urban districts. NAEP has served as an independent monitor of what students 
know and can do in various subject areas, including reading, mathematics, science, US. history, and 
writing. For each subject area measured by NAEP, a framework is used to provide recommendations 
on the content and processes to be assessed, the types of assessment questions to be asked, and the 
administration of the assessment. Each framework is designed to guide the assessment for about a 
decade until it is updated. 


NAEP results, commonly referred to as The Nation’s Report Card, have become an important 
source of information on what US. students know and are able to do in a range of subject areas. The 
resulting data on student knowledge and performance have been accompanied by contextual infor- 
mation that allows analyses of a number of student demographic and instructional factors related 
to achievement. The assessments have been designed to allow comparisons of student performance 
over time and among subgroups of students defined according to region, parental education, gender, 
and race/ethnicity. The Trial Urban District Assessment (TUDA) was initiated in 2002 to report 
on the achievement of public school students in large urban districts. Under this program, NAEP 
has administered its mathematics, reading, science, and writing assessments to samples of students in 
large urban districts that have volunteered to participate in NAEP. 


Resources for Planning the Framework 


Any NAEP framework must be guided by NAEP purposes as well as the policies and procedures 
of the Governing Board, which oversees NAEP. For the NAEP Technology and Engineering Literacy 
Assessment, the main purpose of the framework is to establish what students should know about 
and be able to do with technology and to set forth criteria for the design of the 2014 assessment and 


future assessments. Meeting this purpose requires a framework built around what the communities 
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involved in technology, technology education, educational technology, and technology and engi- 


neering literacy consider to be the knowledge and skills that are most important for NAEP to report. 


In prioritizing the content, the framework developers used the NAEP technological literacy 
steering committee guidelines (summarized later in this chapter). These guidelines recommended 
drawing from the following sources: 


e Existing technology standards from various individual states; 

e National Education Technology Standards by the International Society for Technology in 
Education (ISTE, 2007); 

e Standards for Technological Literacy: Content for the Study of Technology by the Interna- 
tional Technology and Engineering Educators Association (ITEEA, 2007); 

e Framework for 21st Century Learning from the Partnership for 21st Century Skills (P21, 
2007); 

e Influential technology standards from other countries (for example, the United Kingdom); 

© Science Framework for the 2009 National Assessment of Educational Progress (National 
Assessment Governing Board, 2008); 

e Benchmarks for Science Literacy by the American Association for the Advancement of Sci- 
ence (AAAS, 1993); 

e National Science Education Standards from the National Research Council (NRC, 1996); 

e Definitions of technological literacy contained in the No Child Left Behind Act of 2001 
(NCLB, 2001) and the American Recovery and Reinvestment Act of 2009 (ARRA, 
2009); and 

e Research studies and reports, for example, Technically Speaking and Tech Tally: Approaches 
to Assessing Technological Literacy from the National Academy of Engineering (NAE) and 
the National Research Council (NRC); The Intellectual and Policy Foundations of the 21st 
Century Skills Framework (P21, n.d.); and Fostering Learning in the Networked World: The 
Cyberlearning Opportunity and Challenge (NSF, 2008). 


Tables illustrating how the major assessment areas presented in chapter two are aligned with 
these source documents are presented in appendixes C and D. These sources embody a wealth of 
information about technological literacy and technology education. All of them address similar 
issues of K-12 content and assessment, and in many ways they converge on a broad vision of technol- 
ogy and engineering literacy. However, the various documents do not always agree on definitions of 
terms, and in many cases they attach different meanings to phrases such as “educational technology” 
and “technology education,’ which a reader outside the field would find confusing. Consequently, it 
is important to establish clear definitions for the purpose of this framework and the work of NAEP 
that will follow. (See the glossary in appendix A for full definitions of relevant terms used in this 
framework.) 


Definitions of Key Terms Used in This Framework 


Because of the variety of meanings associated with words and terms that are used throughout 
this document, meanings and usages specific to this framework are offered here. The particular 
words and terms at issue are “technology, “engineering,” “technology and engineering literacy,’ and 
“educational technology.” It is important to note that these definitions have been developed for the 
sole purpose of informing NAEP about the field of technology and engineering literacy. No addi- 
tional claim is made regarding the usefulness of these definitions for other purposes—and, in 
particular, they should not be used for the interpretation of state and local assessments, since these 
may be based on different definitions of technology and engineering literacy. This framework 


defines “technology,” “engineering,” and “technology and engineering literacy” as: 


Technology is any modification of the natural or designed world done to fulfill human 


needs or desires. 


Engineering is a systematic and often iterative approach to designing objects, processes, 


and systems to meet human needs and wants. 


Technology and engineering literacy is the capacity to use, understand, and evaluate 
technology as well as to understand technological principles and strategies needed to develop 
solutions and achieve goals. For purposes of this framework, it comprises three areas: Technol- 


ogy and Society, Design and Systems, and Information and Communication Technology. 


The following sections expand and elaborate on the above brief definitions of “technology,” 
“engineering,” and “technology and engineering literacy,’ with the goal of describing the knowledge 
and capabilities that are essential for citizens in the 21st century—and, in particular, in a way that 


can be assessed through an on-demand, large-scale assessment. 


Technology 


Research shows that most Americans associate technology mainly with computers and related 
electronic devices (Cunningham, Lachapelle, & Lindgren-Streicher, 2005). However, while the 
computer is certainly an important example of technology—and one that plays an especially 
important role in this framework—the term “technology” has a broader and deeper meaning. 
Broadly speaking, technology is any modification of the natural or designed world done to fulfill 
human needs or desires. This definition sees technology as encompassing the entire human-made 
world, from the simplest artifacts, such as paper and pencil or a wooden flute, to the most com- 


plex, including the Internet, medical imaging devices, and a country’s entire transportation system. 
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Technology helps people do their jobs: giant particle colliders for physicists, oil paints and canvases 
for artists, and email for just about everyone. Technology also makes leisure time more enjoyable; 

movies, music, and electronic games to entertain; automobiles to get people where they want to go; 
audio and imaging tools to support creative expression; and books (either e-books or the traditional 


paper version) to tell stories of other places and times. 


But technology is not just the products or things that people create. It includes the entire infra- 
structure needed to support the processes used to design, manufacture, operate, and repair tech- 
nological artifacts, from corporate headquarters and engineering schools to manufacturing plants, 
media outlets, and distribution networks (Shakrani & Pearson, 2008). Technology also includes the 
cyberinfrastructure and participatory technologies that open up greater access to complex learning 


and connections of experiences across settings (NSF, 2008). 


Throughout history, technology has been one of the major factors shaping human life and 
human civilization, and, indeed, major periods of human development have typically been identified 
by the dominant technologies of the period: Stone Age, Bronze Age, Iron Age, Industrial Age, and, 
today, the Information Age. Technology itself is constantly changing and evolving, as are its effects 
on society. Ten thousand years ago, humans took the first steps toward agriculture with the pur- 
poseful planting of seeds; 100 years ago, farmers and plant scientists were regularly improving crops 
through hybridization; today, genetic engineering has been harnessed to create specially designed 
crops and farm animals. Perhaps the most dramatic example of technological evolution from today’s 
perspective is the rapid development of communications technology, from the invention of the 
telegraph and telephone in the 19th century to the development of radio, television, and the Inter- 
net in the 20th century—and to the past decade’s explosion of electronic communication and social 
networking (Solomon & Schrum, 2007). Technology has become an enabling force behind global- 
ization, knowledge work, and entrepreneurship (Metiri Group, 2006). With each of these changes 


comes new capabilities—and new challenges. 


Engineering 


Engineering is the process of designing the human-made world. The process typically begins 
with the specifications of needs or wants. Engineers identify constraints, analyze the features of 
systems, and devise plans for developing solutions. Solutions may take the form of artifacts such as 
computer chips or bridges. Solutions may also take the form of improved processes such as assembly 
lines or traffic control. Engineering processes are typically iterative, involving testing and revisions. 
Engineers understand the nature of the technology area to be modified, engage in systems thinking, 


work through engineering design processes, and conduct maintenance and troubleshooting. 


Technology and Engineering Literacy 


Having defined “technology” and “engineering” broadly in these ways, “technology and engi- 
neering literacy” can be defined in an equally broad fashion as the capacity to use, understand, 
and evaluate technology as well as to understand technological principles and strategies needed to 
develop solutions and achieve goals. Technology and engineering literacy, like scientific, mathe- 
matical, or language literacy, is a measure of how well individuals have mastered the processes and 
tools they need to participate intelligently and thoughtfully in the world around them. As described 
in reports from the National Academy of Engineering (NAE, 2006), the International Society for 
Technology in Education (ISTE, 2007), and the International Technology and Engineering Educa- 
tors Association (ITEEA, 2007), technological literacy includes knowledge, capabilities, and critical 
thinking and decision-making skills (McAnear, 2009). From these documents, lists can be extracted 


of what a person who is literate in technology and engineering should know and be able to do. 


Figure 1 depicts the major assessment areas and practices of the NAEP Technology and Engineer- 
ing Literacy Framework. At the center, in blue, are the three areas of technology and engineering lit- 
eracy that will be assessed by NAEP: Technology and Society, Design and Systems, and Information 
and Communication Technology (ICT). The surrounding yellow circle represents three overarching 
types of thinking and reasoning that generalize across the major assessment areas. These crosscutting 
practices will apply across tasks and individual test questions, also referred to as items, in the areas of 
Technology and Society, Design and Systems, and ICT. These key components of the NAEP Tech- 


nology and Engineering Literacy Framework are described on the next page. 


Three Areas of Technology and Engineering Literacy 


In recent decades, students in the United States 
have experienced three quite different (though by no 
means inconsistent) approaches to technology and 
engineering literacy. These are the science, technol- 
ogy, and society approach; the technology education 
approach; and the information and communication 
technology approach. In recognition of the importance, 
educational value, and interdependence of these three 
approaches, this framework includes all three under its 
broad definition of technology and engineering literacy. 


In recognition of the distinct goals and teaching meth- 


ods involved in each, this framework recommends that 


assessment results be reported for each of these areas to Pisutd= Maiotslcaecineta ieGasend Pideivestet 


Technology and Engineering Literacy 
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monitor and analyze the results of each approach over time. The next few paragraphs present a brief 


description of each of these approaches. 


e The Technology and Society assessment area has its roots in the science, technology, and 


society (STS) approach. In 1990 the board of directors of the National Science Teach- 

ers Association defined STS as the “teaching and learning of science and technology in 
the context of human experience” (NSTA, 2006, pp. 229-230). In practice many STS 
programs use societal issues as course organizers, including space travel, insecticide use, 
nutrition, disease, ozone, global warming, and other concerns reported in the popular 
press. Since technological advances and decisions lie at the core of such issues, the focus in 
discussing them is often on the technology involved (Yager & Akcay, 2008). A survey of 
engineering and technology in state science standards found that a majority of state stan- 
dards reflect the STS approach (Koehler, Faraclas, Giblin, Kazerounian, & Moss, 2006). 
The STS approach is represented in this framework under “Technology and Society.” 


The Design and Systems assessment area is partly rooted in the school subject known as 
industrial arts (Dugger, 2005), a popular subject area throughout most of the 20th century, 
which provided education in the use of hand and power tools for fabricating objects from 
wood, metal, or other materials, as well as instruction on industrial processes. As conceived 
today by the field’s professional organization, the International Technology and Engineer- 
ing Educators Association (ITEEA), technological literacy “involves a vision where each 
citizen has a degree of knowledge about the nature, behavior, power, and consequences of 
technology from a broad perspective. Inherently, it involves educational programs where 
learners become engaged in critical thinking as they design and develop products, systems, 
and environments to solve practical problems” (ITEEA, 1996, p. 1). Goals in technology 
education include creating a broad understanding of technology and engineering as well 

as developing specific capabilities in both areas (ITEEA, 2007). A survey of state science 
standards (Koehler et al, 2006) found that many states, especially those in the Northeast, 
include standards consistent with this approach, although not as many as those whose 
standards relate to STS. The engineering design approach is represented in this framework 


under the heading “Design and Systems.” 


Information and communication technology (ICT) is a third approach that has been 
growing in importance over the past three decades. The pervasiveness of technology 

in school, home, work, and play has profound implications for learning in schools and 
throughout life. The field’s major professional organization, the International Society for 


Technology in Education (ISTE), was formed in 1989 by the merger of two associations 


concerned primarily with the use of computers in education. Today, the vision of ICT is 
much broader than the use of computers alone, having expanded from the earlier vision of 
technology as a teaching tool to today’s philosophy of technology as a learning tool. That 
is, the focus is no longer on using technology to assist teachers but rather on giving stu- 
dents new and more powerful ways to gather and assess information, think creatively, solve 
problems, and communicate. As expressed in the society's National Educational Technology 
Standards (ISTE, 2007), ICT includes a variety of student skills that overlap with other 
areas, such as creativity and innovation; communication and collaboration; research and 
information fluency; and critical thinking, problem-solving, and decision-making. These 
skills are applied specifically to the use of digital technologies and media, including the 


Internet and other networking applications. 


Although these information technologies make up one component of technology, broadly 
defined, they have been responsible for many of the most profound changes that have taken place 
in society over the past several decades. And the variety, uses, and power of such information tools 
are expected to grow rapidly over the next decade (The New Media Consortium, 2009). Media, 
telecommunication, and networked technologies are evolving into powerful support systems for 


acquiring skills needed in the 21st century (p. 21, n.d.). 


“Every young person will need to use ICT in many different ways in their adult lives, in order to 
participate fully in a modern society” (Organisation for Economic Co-operation and Development 
[OECD], 2006). The ICT approach to technology and engineering literacy is represented in this 


framework under the heading “Information and Communication Technology.” 


Three Practices 


A person who is literate in technology and engineering should be able to apply “crosscutting 
practices,’ or generalizable ways of thinking, reasoning, and acting that are important across all areas 
of technology and engineering literacy. As depicted in figure 1, these practices are employed within 
and across the three major assessment areas. The practices can be grouped into these three broad 


categories, with several examples of each type of practice: 


Understanding Technological Principles 


e Understands the nature of technology in its broadest sense 
e Is aware of the various digital tools and their appropriateness for different tasks 
e Knows how technology is created and how it shapes society and in turn is shaped by 


society 
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e Understands basic engineering concepts and terms, such as systems, constraints, and 
trade-offs 


e Understands how cultural differences can affect technological choices 


Developing Solutions and Achieving Goals 


e Demonstrates appropriate usage of a wide range of technological tools and systems, ranging 
from kitchen appliances and alarm clocks to cars, computers, communication devices, and 
the Internet 

e Can apply technological concepts and skills creatively, including those of engineering 
design and information technology, to solve problems and meet goals 

e Collects and analyzes data to develop a solution and complete a project 

e Uses multiple processes and diverse perspectives to explore alternative solutions 


e Can evaluate claims and make intelligent decisions 


Communicating and Collaborating 


e Communicates information and ideas effectively to multiple audiences using a variety of 
media and formats 

e Participates thoughtfully and productively in discussing critical societal issues involving 
technology related to humans, the environment, knowledge, and citizenship 


e Collaborates with peers and experts 


The framework recommends that results of the NAEP Technology and Engineering Literacy 
Assessment be reported separately for the three major assessment areas of Technology and Society, 
Design and Systems, and ICT, although it cannot be stressed strongly enough that today’s youth 
are expected to acquire knowledge and skills in all three areas of technology and engineering liter- 
acy. These areas are neither learned separately nor applied separately; they overlap and interact. A 
person who is literate in technology and engineering understands and is able to analyze the rela- 
tionship between technology and society, has a broad understanding of technology and can solve 
problems using the engineering design process, and is able to make fluent use of digital technologies 
and media in creative and innovative ways. Specific assessment targets related to the three areas are 


described at length in chapter two. 


Educational Technology 


Although it is not an assessment target for the purposes of NAEP, the field of educational tech- 
nology provides another example of a common use of the term “technology.” Broadly speaking, the 
field of educational technology is concerned with the use of various types of equipment as teaching 
and learning aids. Many teachers remember when overhead projectors were in widespread use or 


when whiteboards replaced chalkboards. Advocacy for the use of computers in classrooms began 
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more than 20 years ago, and the uses of computers have evolved rapidly from computers-as-teachers 
to computers-as-learning-tools. Today a vast array of computer applications is available for use in all 
school subjects, and these applications are fundamentally altering the way students learn in school, 
giving them unprecedented input into and control of their own learning. Some devices, such as inter- 
active whiteboards, combine technologies for entirely new purposes. An area of digital or cyber liter- 
acy is emerging that encompasses newer forms of technology and media (Kress, 2003; Livingstone, 
van Couvering, and Thumin, 2008). Traditions of media and information literacy are converging and 
focusing on skills needed to take advantage of digital systems for representing and distributing infor- 
mation (Livingstone, 2002). The variety and use of such tools for learning, expression, and commu- 
nication are expected to expand rapidly over the next decade, affecting the way all people—not just 


students—work, collaborate, and communicate (The New Media Consortium, 2009). 


The Relationship Among Science, Technology, Engineering, and Mathematics 


Science, technology, engineering, and mathematics are so closely interlinked that it is often 
difficult to know where one starts and the other ends. Students in science classes are often taught 
about technology, engineering, and mathematics, while students in technology classes learn about 
science, engineering, and mathematics. Technologies are changing fundamentally the ways scientists 
work, and are becoming important components of science education. Students’ skills in using the 
tools of science are becoming components of the “new literacies” (Quellmalz and Haertel, 2008). 

In a recent report on cyber learning, the National Science Foundation points out that research has 
demonstrated that “incorporating information and communications technology into science and 
mathematics can restructure the necessary expertise for reasoning and learning in these domains, 

in effect opening up greater access to complex subject matter.” Examples include multiple linked 
representations in mathematics and modeling and visualizations for understanding and investigating 
complex science (NSF, 2008, p. 13). 


For the purposes of designing a framework to assess technology and engineering literacy, it 
is important to keep the distinctions among the science, technology, and engineering clear. The 
relationship among engineering, science, and technology is explained this way in the joint National 


Academy of Engineering/National Research Council publication Technically Speaking: 


Science and technology are tightly coupled. A scientific understanding of the natural world 
is the basis for much of technological development today. The design of computer chips, 
for instance, depends on a detailed understanding of the electrical properties of silicon and 
other materials. The design of a drug to fight a specific disease is made possible by knowl- 


edge of how proteins and other biological molecules are structured and how they interact. 
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Conversely, technology is the basis for a good part of scientific research. The climate models 
meteorologists use to study global warming require supercomputers to run the simulations. 
And like most of us, scientists in all fields depend on the telephone, the Internet, and jet 
travel (NAE & NRC, 2002, pp. 13-14). 


One other distinction that is important to make is between technology and engineering. Again 


the explanation from Technically Speaking is helpful: 


Technology is a product of engineering and science, the study of the natural world. Science 
has two parts: (1) a body of knowledge that has been accumulated over time, and (2) a 
process—scientific inquiry—that generates knowledge about the natural world. Engineering, 
too, consists of a body of knowledge—in this case, knowledge of the design and creation of 
human-made products—and a process for solving problems (NAE & NRC, 2002, p. 13). 


Of the three terms—science, technology, and engineering—the clearest parallel is between sci- 
ence and engineering, as both represent an approach to knowledge taken by a group of well-trained 
professionals. As explained in the National Science Education Standards (NRC, 1996, p. 166), “Sci- 
entists propose explanations for questions about the natural world, and engineers propose solutions 


relating to human problems, needs, and aspirations.” 


A fourth area that is often associated with these other three is mathematics. Although math- 
ematics is a field in its own right, distinct from science and engineering, mathematical tools are 
essential to the work of both scientists and engineers. In fact, science, technology, engineering, and 


mathematics are so intimately connected that they are frequently referred to by the acronym STEM. 


The Framework Development Process 


In October 2008, the Governing Board awarded a contract to WestEd to develop a framework 
and specifications for assessing technological literacy. In carrying out its work, WestEd collaborated 
with the Council of Chief State School Officers (CCSSO), ISTE, ITEEA, the Partnership for 21st 
Century Skills, and the State Educational Technology Directors Association (SETDA). In working 
with these groups, WestEd used a process for developing the framework and related products that 


was inclusive, deliberate, and designed to achieve as much broad-based input as possible. 


A two-tiered committee structure with a steering committee anda planning committee pro- 
vided the expertise to develop the framework as specified by the Governing Board. (See “NAEP 
Technology and Engineering Literacy Project Committees and Staff” section for lists of commit- 


tee members.) The two committees were composed of members who were diverse in terms of role, 
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gender, race or ethnicity, region of the country, perspective, and expertise regarding the content of 


the assessment to be developed. 


Summary of the Steering Committee Guidelines 


The following definition of technology and engineering literacy should be used: Technol- 
ogy and engineering literacy is the capacity to use, understand, and evaluate technology as 
well as to understand technological principles and strategies needed to develop solutions 
and achieve goals. The steering committee also suggested that the planning committee 
examine other definitions of technology and engineering literacy, especially those used by 
the states. 

The assessment should consist of technological content areas, to be reported as scale 


scores, and technological practices that characterize the field. The content areas must 


include Design and Systems, Information and Communication Technology (ICT), and 


Technology and Society. 

Three practices should be addressed in the assessment: Understanding Technologi- 

cal Principles, Developing Solutions and Achieving Goals, and Communicating and 
Collaborating. 

The content and context for the assessment should be informed by state standards and 
assessments, national and international standards, and research. 

Assessment tasks and items should minimize the need for prior knowledge of specific 
technologies other than ICT; tasks should relate to real-world problems and contexts. 

As far as possible, life situations familiar to students and local and contemporaneous con- 
ditions should be used to design tasks as a way to confer relevance to each grade level being 
assessed. 

The assessment should focus on a broad base of knowledge and skills, not on specific tech- 
nological processes that may change. 

The assessment should use innovative computer-based strategies that are informed by 
research on learning and are related to the assessment targets. 

The computer-based assessment strategies should be informed by what is known about all 
learners, including English language learners (ELL) and students with disabilities. 

All the items and tasks included as exemplars in the framework and specifications docu- 


ments should have feasible responses that can be awarded score points. 


The steering committee members included leaders from a variety of fields and subject areas, 
including schools, engineering, education, 21st-century skills, the Internet, business, science educa- 
tion, general education, and assessment. The co-chairs were balanced, with one representing tech- 


nology in schools, and the other, STEM and assessment. Functioning as a policy and oversight body, 
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this group outlined the planning committee’s responsibilities in developing the framework. The 
committee reviewed and provided feedback on drafts of the framework and related materials. The 


interaction between the two committees was iterative over the course of the project. 


The planning committee, as supported by the project staff, was the development and produc- 
tion group responsible for drafting the framework, specifications, recommendations for contextual 
variables, and preliminary technology and engineering literacy achievement level definitions. This 
committee was composed of business leaders, researchers, state and district technology coordina- 
tors, teachers, and representatives from educational organizations as well as experts in research, 
assessment, and evaluation. As with the steering committee, the planning committee co-chairs were 
balanced, with one being an expert in ICT-based learning and assessment and the other being an 


expert in K-12 science and engineering education. 


The planning committee’s work was guided by policies, goals, and principles identified by the 
steering committee (see summary box on the previous page; for full text, see appendix B). In addi- 
tion to the sources cited previously, the planning committee relied on guidance provided by NAEP 
Technological Literacy Framework and Specifications Development: Issues and Recommendations, a 


paper prepared by Sharif M. Shakrani and Greg Pearson for the Governing Board. 


The work was carried out in a series of meetings, with numerous telephone calls and email 
exchanges between meetings. From December 2008 through September 2009, the steering commit- 
tee met four times and the planning committee met six times. Three steering committee and plan- 
ning committee meetings overlapped so that the two committees could share understandings and 
discuss critical issues. Governing Board staff supported and participated in all of the meetings. In 
addition, between formal work sessions, Governing Board members and staff provided feedback and 


guidance on project documents and processes. 


After the development of initial drafts of the framework, WestEd led a series of outreach efforts 
to solicit feedback. Formal activities included, but were not limited to, presentations and sessions 
with industry representatives (for example, IBM and Cisco), ITEEA, the Organisation for Eco- 
nomic Co-Operation and Development (OECD) International ICT Research Workshop, CCSSO, 
SETDA, ISTE, and the Partnership for 21st Century Skills Webinar. (See NAEP Technology and 
Engineering Literacy Project Committees and Staff section for more complete lists of individuals 
and organizations that contributed to the development of this framework.) The planning commit- 
tee reviewed feedback from these groups as well as from the steering committee and made revisions, 
as it deemed appropriate. Governing Board members and staff closely monitored the project and 
provided comments on draft documents. After final approval from the steering committee, the 


framework was submitted to the Governing Board for action. 
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Challenges of Developing the NAEP Technology and Engineering Literacy Framework 
There were a number of challenges in developing the NAEP Technology and Engineering Liter- 


acy Framework that were not necessarily encountered in developing other NAEP frameworks. These 
included: (1) the newness of the endeavor, (2) diffuse curricula, (3) varying definitions, (4) measure- 
ment constraints, (5) time and resource constraints, (6) designing an entirely computer-based assess- 


ment, and (7) predicting future changes in technology. Each of these challenges is discussed below. 


Newness of the Endeavor 


Technology and engineering literacy is a growing and evolving area. Unlike other NAEP sub- 
jects, such as reading or mathematics, there is no existing NAEP framework to draw on. Moreover, 
the existing item banks in the United States and other countries are very limited (NAE, 2006). The 
technology and engineering literacy staff and committee members obtained only a limited number 
of sample items from outside sources, reflecting the immature state of assessing technology and 


engineering literacy. 


Diffuse Curriculum 


Unlike science and mathematics, which have a sequential curriculum taught by subject-area 
specialists in high school or by generalists in elementary school, technology and engineering educa- 
tion as a whole does not have a unified scope and sequence. Some individual courses (for example, 
science, technology, and society; pre-engineering; and computer modeling) are likely to follow state 
standards and have a specified curriculum with a scope and sequence, but these individual courses 
are generally not grouped together under the rubric of technology courses. When they are, the 
courses under such a heading may vary from place to place. ICT has also been integrated into the 
curriculum in a variety of ways. While ICT learning is often infused into existing core subjects, it 
is not always assessed and reported as part of these subjects. In addition, there is not a clear scope 
and sequence for ICT knowledge and skills, either as a stand-alone curriculum or integrated into 
core subjects, which may result in an inconsistent application of technology literacy standards 
across different grades, subjects, and states. As mentioned earlier, all teachers have a role in teaching 
technology, so in most cases, teachers are not singled out as technology teachers in the same way 
that, for example, mathematics or history teachers are identified with those subject areas. The result 
(and implication for this assessment) is that the specific technology concepts and practices to which 
students have been exposed are hit-and-miss and mostly unknown. Students can say what mathe- 
matics or science courses they have taken, but specifying the range of their education in technology 


and engineering and their use is more ambiguous. 
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Varying Definitions 


One of the most debated issues in developing the framework was the definition of “technology 
and engineering literacy,’ as different definitions abound. Indeed, even the terms are not agreed 
upon, as some organizations refer to “technological literacy” while others refer to “technology liter- 
acy” or “engineering literacy.” In this report, for consistency, the term “technology and engineering 
literacy” will be used throughout while recognizing that this terminology differs from what is used 
by some groups. ITEEA, the National Research Council, and ISTE have definitions of technol- 
ogy and engineering literacy. Meanwhile, the federal No Child Left Behind Act of 2001 (NCLB) 
required that every student be “technologically literate by the time the student finishes the eighth 
grade,” but the law itself is vague in defining what technological literacy is. States have therefore had 
flexibility in determining what technology and engineering literacy means and how it should be 


assessed. 


Many states have adopted a common definition worked out by SETDA in 2002, which states, 
“Technology literacy is the ability to responsibly use appropriate technology to communicate, solve 
problems, and access, manage, integrate, evaluate, and create information to improve learning in all 
subject areas and to acquire lifelong knowledge and skills in the 21st century.” The federal American 
Recovery and Reinvestment Act of 2009 adds real-world consequences to this shared definition by 
providing grants to state and local agencies and schools based on their abilities to meet goals defined 


by long-range educational technology plans, most of which include this definition. 


The NAEP Technology and Engineering Literacy Framework attempts to unify the concepts 
and skills presented in these and other definitions under one umbrella definition. The definition 
presented earlier in this chapter is used only to understand the results of the NAEP assessment in 
technology and engineering literacy. While this framework and subsequent NAEP results may be 
informative to education administrators, policyholders, industry and business leaders, and the gen- 
eral public, the definition of technology and engineering literacy presented here should not be used 
to interpret results from other assessments used at state and local levels. To further distinguish the 
NAEP Technology and Engineering Literacy Assessment from other technology and engineering 
literacy assessments, the project committees have recommended that the results be reported in terms 
of three individual scores, each reflecting performance in one of the three main areas of technology 
and engineering literacy: Technology and Society, Design and Systems, and Information and Com- 


munication Technology. An overall composite score will also be reported. 
Measurement Constraints 


NAEP, like any large-scale assessment in education, the workplace, or clinical practice, is 


constrained in what it can measure. This has implications for the proper interpretation of NAEP 
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Technology and Engineering Literacy Assessment results. The framework is an assessment frame- 
work, not a curriculum framework. Although the two are clearly related, each has a different pur- 
pose and a different set of underlying assumptions. A curriculum framework is designed to inform 
instruction, to guide what is taught, and, often, to guide how it is taught. It represents a wide uni- 
verse of learning outcomes from which educators pick and choose what and how they teach. An 
assessment framework is a subset of the achievement universe from which assessment developers 
must choose to develop sets of items that can be assessed within time and resource constraints. 
Hence, the content to be assessed by NAEP has been identified as that considered central to tech- 
nology and engineering literacy. 


Asa result, some important outcomes of technology and engineering literacy (broadly defined) 
that are valued by general educators, engineers, teachers of technology, and the business community 
but that are difficult and time-consuming to measure—such as habits of mind, sustained projects, 
and collaboration—will be only partially represented in the framework and on the NAEP Technol- 
ogy and Engineering Literacy Assessment. Moreover, the wide range of technology and engineering 
standards in the guiding national documents that were incorporated into the framework had to 
be reduced in number to allow some in-depth probing of fundamental knowledge and skills. As a 
result, the framework and the specifications represent a distillation rather than a complete represen- 
tation of the original universe of achievement outcomes specified by technology and engineering 


education documents. 


Time and Resource Constraints 


Time and resources limit what NAEP can assess. Like most standardized assessments, NAEP is 
an “on-demand” assessment. That is, it is given as a scheduled event outside the normal classroom 
routine with uniform conditions for all of the students being assessed. In particular, NAEP has 
a limited amount of time—in this case, approximately one hour per student—to ascertain what 
students know and can do. However, standards presented by professional associations and the states 
contain goals that require an extended amount of time (days, weeks, or months) to assess. To assess 
the achievement of students in the kinds of extended activities that are a central feature of these 
other standards and of many curricula, it would be necessary to know a number of things about the 


students, including their: 


e Reasoning while framing their goals; 

e Planning for projects and the implementation of the plan; 

e Skills in using technologies to gather, manage, and analyze data and information related to 
project goals; 

e Capabilities to meet unpredictable challenges that arise during actual, ongoing prob- 


lem-solving and achievement of goals; 
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e Lines of argument in deciding how to alter their approaches in the light of new evidence; 
e Engagement with peers and experts in addressing goals and deciding how to achieve them; 
and 


e Deliberations and reasoning when evaluating progress, trade-offs, and results. 


NAEP, like other on-demand assessments, then, cannot be used to draw conclusions about stu- 
dent achievement with respect to the full range of goals of technology education, broadly defined. 
States, districts, schools, and teachers can supplement NAEP and other standardized assessments 
to assess the full range of education standards that address technology and engineering literacy. In 
addition to describing the content and format of an examination, assessment frameworks, like this 


one, signal to the public and to teachers some core elements of a subject that are important. 


Designing a C omputer-Based Assessment 


Although some NAEP assessments (the 2009 Science Assessment, for example) have called for 
interactive computer tasks, as of 2009, only the NAEP Writing Assessment has been totally com- 


puter-based. The design challenges of creating such an assessment include: 


e Developing the requisite number of tasks and items (test questions), especially since so few 
tasks and items exist that can serve as samples. 

e Constructing tasks and items that provide whatever prior knowledge is required to answer 
the question. As so many contexts are available in which to set items, developers cannot 
assume that students will have prior knowledge of the specific topics (for example, core 
subjects such as the humanities or mathematics) or technologies (for example, transpor- 
tation, health, or electronics) within the context. Items must not require students to have 
prior knowledge of specific technologies, and the knowledge required about particular 
technologies must be presented in the item. 

e Determining the features and functions of the complete tools students will use. 

e Determining what aspects of student responses to an item need to be assessed. Are the 
attempts a student makes while trying out a design or using a simulation important to cap- 
ture? What about the pathway the student follows or the number of mistakes made before 
getting a correct answer? Rather than a single question and answer, an item might have 


several components that are assessed. 


In addition to the issues above, there will also be administrative challenges, such as whose com- 
puters the students use to complete the assessment, handling students’ different levels of access to 
computer technology, and contingencies in case equipment malfunctions. The framework designers 
were aware of these factors when developing the framework, but they focused on the design factors, 
leaving the challenge of determining how best to administer the NAEP Technology and Engineer- 


ing Literacy Assessment to those involved in the assessment development phase. 
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Predicting Future Changes in Technology 


The framework attempts to strike a balance between what can reasonably be predicted about 
future technology and engineering literacy education and what students are likely to encounter in 
their curriculum and instruction now and over the next decade. For example, specific communica- 
tion technologies in use today (Internet-connected multimedia smartphones and personal digital 
assistants [PDAs]) would not have been familiar to students a decade ago and may well be obsolete a 


decade from now. 


The framework is intended to be both forward-looking (in terms of what technology content 
and usage will be of central importance in the future) and reflective (in terms of current technol- 
ogy). Because it is impossible to predict with certainty the shape of educational technology and 
technology education, the choices made for each assessment administration should be revisited in 


response to future developments. 


It is a significant challenge to write a framework for the future, and the challenge is especially 


great for the subject of technology and engineering literacy. 


Overview of Framework Chapters 


The following text describes the content of the remaining four chapters of the 2018 NAEP Tech- 
nology and Engineering Literacy Framework. 


Chapter Two: Areas of Technology and Engineering Literacy 


This core chapter identifies the assessment targets for the 2018 assessment of technology and 
engineering literacy. The targets are grouped into the three major areas: Technology and Society, 
Design and Systems, and Information and Communication Technology (ICT). Each of the major 
areas is broken down into subareas. Each subarea has a listing of key principles and a chart identify- 
ing what students should know and be able to do at grades 4, 8, and 12. Each subarea has between 6 


and 15 assessment targets. 


Chapter Three: Practices and Contexts for Technology and Engineering Literacy 


Chapter three has two major purposes. First, it articulates the kinds of thinking and reason- 
ing that students are expected to demonstrate when responding to the assessment tasks and items. 
Three practices are presented: (1) understanding technological principles, (2) developing solutions 
and achieving goals, and (3) communicating and collaborating. Each practice is applied to the 


three major technology and engineering literacy areas in a chart. There are also tables that apply 
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the practices to selected principles and to the subareas. Illustrative tasks and items suggest how the 


practices can be represented along with targets in each of the areas and subareas. 


Next, the chapter describes the contexts for the assessment—that is, the core school subjects and 
areas of technology, such as humanities, social sciences, medical imaging, publishing, or recycling, 
that will serve as backdrops for assessment questions. The choice and presentation of contexts are 
important because the framework cannot assume that students have prior knowledge of a specific 
topic or technology. If information about a specific type of technology is needed to respond to 
an item, this information must be provided to students in the item as contextual detail. Potential 
contexts are discussed within the three major areas: Technology and Society, Design and Systems, 
and Information and Communication Technology. Each of the three areas has a chart that provides 
examples of how different contexts can be used to formulate tasks and items in each of the three 


assessment areas. 


Chapter Four: Overview of the Assessment Design 


This chapter is an overview of the major components of the NAEP Technology and Engineering 
Literacy Assessment. It discusses the types of tasks and items to which students will respond. These 
will be scenario-based items and discrete items. Examples illustrate how students might respond and 
what parts of their response would be assessed. Balance in four ways is an important requirement in 
the assessment: balance by major assessment areas and grades, balance by technological practices and 
grades, balance by assessment set types and grades, and balance by response types and grades. The 
chapter has charts containing percentages of testing time recommended for each of these features. 
The chapter concludes with suggestions for universal design and adaptations for students with dis- 


abilities, and English language learners. 


Chapter Five: Reporting Results of the NAEP Technology and Engineering Literacy 
Assessment 


The last chapter of the framework discusses the issues involved in reporting data from the 
Technology and Engineering Literacy Assessment. Topics include NAEP sampling techniques, how 
NAEP results are reported, reporting scale scores and achievement levels, and reporting contextual 
variables. Appropriate and inappropriate uses of NAEP reporting are also discussed, especially in 
light of the varying definitions of technology and engineering literacy. 
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CHAPTER TWO 


AREAS OF TECHNOLOGY AND ENGINEERING LITERACY 


Introduction 


This chapter describes the essential knowledge and capabilities that will be assessed on the 2018 
NAEP Technology and Engineering Literacy Assessment. Although it is not possible to assess 
every aspect of technology and engineering literacy, this framework identifies a set of assessment 
targets related to the nature, processes, and uses of technology and engineering that are essential for 
21st-century citizens. The assessment targets are organized into three major areas: Technology and 


Society, Design and Systems, and Information and Communication Technology. 


These three areas of technology and engineering 
literacy are interconnected. The relationship among 
these three major assessment areas can be illus- 
trated as a three-sided pyramid in which each side 
supports the other two. For example, to address 

an issue related to technology and society, such as 
clean water, energy needs, or information research, a 
person who is literate in technology and engineering 
must understand technological systems and the en- 


gineering design process and be able to use various 
information and communication technologies to re- 


search the problem and develop possible solutions. 


This chapter provides descriptions of each of the major areas of technology and engineering 
literacy as well as subareas and tables of assessment targets that specify what students in grades 4, 8, 
and 12 should know and be able to do. The assessment targets describing what students should be 
able to do foreshadow the crosscutting “practices” —ways of thinking and reasoning—described in 


chapter three. 
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It will be apparent when reading assessment targets across the grade-level rows that learning is 
cumulative—that is, later grades build on what has been learned in earlier grades, so that students 
develop a greater sophistication and depth of understanding as they advance in school. For instance, 
elementary school students think of a technological change in terms of a succession of “products,” 
as in the evolution of writing technology from clay tablets to pen and paper and to computers and 
printers. Middle school students are able to think in terms of technological “processes,” such as the 
processing of food, or the extraction of metal from ore. High school students have learned to think 
in terms of technological “systems,” such as a city’s public transportation system or water purification 
system. 


There is some overlap among the three major assessment areas. For example, there may be refer- 
ences to Information and Communication Technology (ICT) and to Design and Systems within 
the Technology and Society area. This is due in part to the mutual support that these technological 
principles and skills lend to each other, and it serves to emphasize that individuals who are literate in 
technology and engineering can bring these ways of thinking and acting to bear on any problem or 
goal. Thus, ICT knowledge and skills are called on during the design of technologies; information 
and communication technologies are developed by engineering design processes; and the myriad 
technologies designed to meet human needs, including the ubiquitous information and communi- 


cation technologies, are influenced by and have impacts on society. 


The three major assessment areas of technology and engineering literacy and their correspond- 


ing subareas are presented in table 2.1 and briefly summarized below. 


Table 2.1 Major areas and subareas of 2018 NAEP Technology and 
Engineering Literacy Assessment 


Technology and Society Design and Systems Information and 
Communication Technology 
(ICT) 
A. Interaction of Technol- A. Nature of Technology A. Construction and 
ogy and Humans B. Engineering Design Exchange of Ideas and 
B. Effects of Technologyon | C. Systems Thinking Solutions 
the Natural World D. Maintenance and B. Information Research 
C. Effects of Technology on Troubleshooting C. Investigation of Problems 
the World of Information D. Acknowledgment of 
and Knowledge Ideas and Information 
D. Ethics, Equity, and E. Selection and Use of 
Responsibility Digital Tools 


| 22 


Technology and Society 


A. Interaction of Technology and Humans concerns the ways society drives the improvement and 


creation of new technologies, and how technologies serve society as well as change it. 


B. Effects of Technology on the Natural World is about the positive and negative ways that tech- 


nologies affect the natural world. 


C. Effects of Technology on the World of Information and Knowledge focuses on the rapidly 
expanding and changing ways that information and communications technologies enable data to be 
stored, organized, and accessed and on how those changes bring about benefits and challenges for 


society. 


D. Ethics, Equity, and Responsibility concerns the profound effects that technologies have on 
people, how those effects can widen or narrow disparities, and the responsibility that people have for 


the societal consequences of their technological decisions. 


Design and Systems 
A. Nature of Technology offers a broad definition of technology as consisting of all the products, 
processes, and systems created by people to meet human needs and desires. 


B. Engineering Design is a systematic approach to creating solutions to technological problems and 


finding ways to meet people’s needs and desires. 


C. Systems Thinking is a way of thinking about devices and situations so as to better understand 


interactions among components, root causes of problems, and the consequences of various solutions. 


D. Maintenance and Troubleshooting is the set of methods used to prevent technological devices 


and systems from breaking down and to diagnose and fix them when they fail. 


Information and Communication Technology (ICT) 


A. Construction and Exchange of Ideas and Solutions concerns an essential set of skills needed 
for using Information and Communication Technology (ICT) and media to communicate ideas 
and collaborate with others. 


B. Information Research includes the capability to employ technologies and media to find, evalu- 
ate, analyze, organize, and synthesize information from different sources. 


C. Investigation of Problems concerns the use of Information and Communication Technology to 


define and solve problems in core school subjects and in practical situations. 


D. Acknowledgment of Ideas and Information involves respect for the intellectual property of 
others and knowledge of how to credit others’ contributions appropriately, paying special attention 


to the misuse of information enabled by rapid technological advances. 
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E. Selection and Use of Digital Tools includes both knowledge and skills for choosing appropriate 


tools and using a wide variety of electronic devices, including networked computing and communi- 


cation technology and media. 


Resources Used in the Development of Assessment Targets 


The process of developing the assessment targets drew heavily on documents created over the 


past two decades by national experts in a wide variety of fields. Primary source documents include: 


Benchmarks for Science Literacy (AAAS, 1993); 

Engineering in K-12 Education (NRC, 2009); 

Framework for 21st Century Learning (Partnership for 21st Century Skills, 2007); 
National Educational Technology Standards (ISTE, 2007); 

National Science Education Standards (NRC, 1996); 

Science for All Americans (AAAS, 1989); 

Standards for Technological Literacy (ITEA, 2002); 

Standards for the 21st Century Learner (American Association of School Librarians, 2007); 
Tech Tally: Approaches to Assessing Technological Literacy (NAE, 2006); and 

Technically Speaking: Why All Americans Need to Learn More About Technology (NRC & 
NAE, 2002). 


Other documents that focused on science but recognized the importance of knowledge and 


skills in technology were valuable resources as well. These included: 


The Assessment & Teaching of 21st Century Skills, a Cisco/Intel/Microsoft project; 
Benchmarking for Success: Ensuring U.S. Students Receive a World Class Education 
(National Governors Association, Council of Chief State School Officers, and Achieve, 
2008); 

Best practices in various state frameworks on science, technology, and engineering; 

Key Competencies for Lifelong Learning: European Reference Framework (European Com- 
munities, 2007); 

PISA 2006: Science Competencies for Tomorrow's World (Organisation for Economic 
Co-operation and Development, 2007); 

Science Framework for the 2009 National Assessment of Educational Progress (National 
Assessment Governing Board, 2008); and 

Trends in International Mathematics and Science Study (TIMSS) (National Center for 
Education Statistics, 2008). 


The steering and planning committees recognize and appreciate the efforts of Achieve, Inc., the 


American Association for the Advancement of Science, the International Society for Technology 


in Education, the International Technology and Engineering Educators Association, the National 


Academy of Engineering, the National Research Council, and the Partnership for 21st Century 
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Skills for their efforts in developing these source materials, for giving permission to quote the mate- 


rials when desired, and for assisting in developing this framework. 


Technology and Society 


From the beginning of human culture, technology and 
society have been closely intertwined. From stone tools to 
computers and the Internet, technologies have allowed people 
to shape the physical world and the world of knowledge to 


meet their needs and wants, to extend the reach of their bodies, 


hands, and minds, to span rivers, and to traverse continents. 
From arrowheads to communication devices, technologies have 


always been an intrinsic part of civilization, and this is particu- 


larly true today, in the early part of the 21st century. This rela- 
tionship is reflected in all of the national standards documents 
reviewed for this framework. It follows that awareness of the relationship between technology and 


society is an essential aspect of technology and engineering literacy. 


Essential knowledge and skills for this facet of technology and engineering literacy are divided 
into four subareas: 


A. Interaction of Technology and Humans; 

B. Effects of Technology on the Natural World; 

C. Effects of Technology on the World of Information and Knowledge; and 
D. Ethics, Equity, and Responsibility. 


A fundamental principle in the area of Interaction of Technology and Humans is that societ- 
ies shape the technologies that are developed and used, and that those technologies in turn shape 
societies. Students are expected to demonstrate their understanding of the positive and negative 
effects that technologies may have on different aspects of society, as well as their capability to ana- 
lyze historical and current examples of the technology-society relationship using concepts such as 
criteria, constraints, trade-offs, and consequences. Students should weigh societal and behavioral 
changes along with purely technological solutions. For example, encouraging the recycling and reuse 


of household materials may be more cost-effective than building new waste facilities. 


Effects of Technology on the Natural World takes a nuanced view of the relationship between 
technology and environmental change, recognizing both the negative impacts of technology on 
the environment and the ways in which people have used technology to restore and protect natural 


environments. Students are expected to recognize that technological decisions involve competing 
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priorities and also to consider the consequences of alternative decisions in developing sustainable 


solutions to environmental problems. 


Effects of Technology on the World of Information and Knowledge addresses the increasing 
access permitted by information technology to expertise and information, the many powerful meth- 
ods for storage and management of information, the expansion of the capability to express ideas and 


representations of dynamic phenomena, and the support of distributed teamwork. 


The area of Ethics, Equity, and Responsibility addresses one of the most important aspects of 
technology and engineering literacy: the fact that technological decisions made by some people 
have significant impacts on others. Many of the thorniest technological issues in society concern 
effects that cross borders, such as acid rain, and many of them have global implications, such as the 
attribution and ownership of ideas and products and the effects of fossil fuel use on climate. The 
focus of this area is on general principles that can be applied when thinking about ethical issues that 
concern various technologies, although individual assessment items will inevitably tend to focus on 
a specific technology. The framework identifies the knowledge and skills that students should have 
for analyzing the issues, gathering evidence that could support multiple perspectives, and presenting 
alternative solutions to technological issues that have ethical implications. The framework does not 


take positions on controversial issues. 


The following narrative provides an overview of each subarea, followed by tables that detail the 
knowledge and skills that will be assessed by the 2018 NAEP Technology and Engineering Literacy 
Assessment in the area of Technology and Society. 


A. Interaction of Technology and Humans 


Many students are first exposed to the interaction between technology and human society 
through the study of history. They learn about the “ages of civilization,” starting with the Stone Age, 
the Bronze Age, the Iron Age, the Industrial Age, and, most recently, the Information Age. So these 
students have already been provided with a number of examples of how societies meet their needs by 
transforming the natural materials in the world around them to create new technologies, and they 
have seen how these technologies in turn shape their societies and their relationship to other societ- 


ies through such mechanisms as trade, communication, war, and assimilation. 
Students are also expected to know from history and from their personal experiences that the 


relationship between technology and society is reciprocal. Society drives technological change, 


while changing technologies, in turn, shape society. 
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Although the effects of technological change are more difficult to discern when the period of 
observation is a few years rather than a number of centuries, students are still capable of reflecting 
on the technological changes that have occurred during their lifetimes. They should also be able to 
observe how the technological changes that are underway are driven by the needs of society, and 
they should be able to predict what some of the consequences of those new technologies might be. 
Examples of technological changes that nearly all students will have observed include new kinds 
of media, computers, and communication systems; the development of more fuel-efficient cars; 
the construction of new or improved buildings, roads, and bridges; and new foods and types of 


clothing. 


Key principles in the area of Interaction of Technology and Humans that all students can be 


expected to understand at increasing levels of sophistication are: 


e The relationship between technology and society is reciprocal. Society drives technological 
change, while changing technologies in turn shape society. 

e Technological decisions should take into account both costs and benefits. 

e When considering technological decisions that involve competing priorities, it is helpful to 
consider the trade-offs among alternative solutions. 

e Technologies may have unanticipated consequences, which become apparent only over 
time as the technology becomes more pervasive or powerful. 

e Technological solutions are developed and evaluated on the basis of criteria and 


constraints. 


Fourth graders are expected to know that people’s needs and desires determine which technolo- 
gies are developed or improved. For example, cellphones were invented, produced, and sold because 
people found it useful to be able to communicate with others wherever they were. Students should 
also know that these new products, tools, and machines in turn affect the lives of individuals, fami- 
lies, and whole communities. An example is how transportation and communications systems enable 


people who live far apart to work together and interact with each other in new ways. 


Eighth graders are expected to understand how technologies and societies coevolve over signif- 
icant periods of time. For example, the need to move goods and people across distances prompted 
the development of a long series of transportation systems from horses and wagons to cars and 
airplanes. They should also recognize that technologies may have effects that were not anticipated 
and that do not become apparent until the technology becomes widespread. For example, it was 
not until cellphones were widely used that it became apparent that people would use them while 
driving, creating an increased risk of accidents. Finally, eighth graders should be able to compare the 


effects of the same technology on different societies. 
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Twelfth graders are expected to realize that the interplay between culture and technology is 
dynamic, with some changes happening slowly and others very rapidly. They should be able to use 
various principles of technology design—such as the concepts of trade-offs and unintended con- 
sequences—to analyze complex issues at the interface of technology and society, and to consider 
the implications of alternative solutions. For example, modern medical technologies have greatly 
increased average lifespan, but many of these new technologies are very costly and not available to 
everyone. The availability of advanced medical technologies also differs from country to country as a 


result of differing economic and political conditions and cultural values. 
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Table 2.2 Interaction of Technology and Humans assessment targets 


for grades 4, 8, and 12 


A. Interaction of Technology and Humans 


Fourth graders should be aware of how products, tools, and machines affect communities and make it possible 
for people to work together. Eighth graders should understand how society drives technological change and 
how new or improved technologies affect a society’s economy, politics, and culture. Twelfth graders should have 
a heightened cultural sensitivity and attain a global view of the interplay between technology and culture. 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

T.4.1: People’s needs and desires 
determine which new tools, prod- 
ucts, and machines are developed 
and made available. 


T.4.2: The introduction of a new 
tool, product, or machine usually 
brings both benefits and costs, and 
it may change how people live and 
work. 


Students know that: 

T.8.1: Economic, political, social, 
and cultural aspects of society drive 
improvements in technological 
products, processes, and systems. 


T.8.2: Technology interacts with 
society, sometimes bringing about 
changes ina society's economy, pol- 
itics, and culture, and often leading 
to the creation of new needs and 


wants. 


Students know that: 

T.12.1: The decision to develop a 
new technology is influenced by 
societal opinions and demands. 
These driving forces differ from 


culture to culture. 


T.12.2: Changes caused by the 
introduction and use of a new tech- 
nology can range from gradual to 
rapid and from subtle to obvious, 
and can change over time. These 
changes may vary from society to 
society as a result of differences in 

a society's economy, politics, and 
culture. 


Students are able to: 

T.4.3: Identify potential positive 
and negative effects of the intro- 
duction of a new technology into a 
community. 


Students are able to: 

T.8.3: Describe and analyze posi- 
tive and negative impacts on society 
from the introduction of a new or 
improved technology, including 
both expected and unanticipated 
effects. 


Students are able to: 

T.12.3: Choose an appropriate 
technology to help solve a given 
societal problem, and justify the 
selection based on an analysis of 
criteria and constraints, available 
resources, likely trade-offs, and rel- 
evant environmental and cultural 


concerns. 


T.4.4: Compare the effects of two 
different technologies on their 
own lives by imagining what their 
lives would be like without those 
technologies. 


T.8.4: Compare the impacts of 

a given technology on different 
societies, noting factors that may 
make a technology appropriate and 
sustainable in one society but not 
in another. 


1.12.4: Analyze cultural, social, 
economic, or political changes 
(separately or together) that may be 
triggered by the transfer of a spe- 
cific technology from one society 
to another. Include both antici- 
pated and unanticipated effects. 
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B. Effects of Technology on the Natural World 


As with technology’ influences on culture and society, the effects of a technology on the envi- 
ronment can be either positive or negative. Since the Industrial Revolution and the rapid growth of 
human populations, the potential for technology to have a major impact on the environment has 
grown. Consequently, an essential aspect of technology and engineering literacy is an understand- 
ing of certain key principles about the effects of technology on the natural environment and of the 
many important efforts that people have made to preserve natural habitats, reduce air and water 


pollution, and maintain a healthful environment. 


Individuals who are literate in technology and engineering should be aware of methods that 
have been developed to reduce the environmental impacts of technology. For example, an important 
step in designing a new product is to take the product’s life cycle into account. Such an analysis may 
start with the raw materials that need to be mined or grown, the industrial processes and energy 
needed to manufacture the product, the transportation technologies required to get it to market, 


and its eventual disposal when the product is no longer needed. 


Other ways to reduce environmental impact include the use of communication technologies 
to allow people to work at home rather than physically commute, the use of computer models to 
optimize industrial processes to conserve energy and reduce waste, and the expansion of alternative 


energy sources such as wind power. 


In finding a balance between technological development and environmental protection, a key 
overarching principle is that one should attempt, when possible, to find sustainable solutions. As 
defined by the Brundtland Commission in 1987, “sustainable solutions” are those that meet the 
needs of the present without compromising the ability of future generations to meet their own 


needs. 


Key principles in the area of Effects of Technology on the Natural World that all students 


can be expected to understand at increasing levels of sophistication are: 


e The use of technology may affect the environment positively or negatively. 

e Some technological decisions put environmental and economic concerns in competi- 
tion with one another, while others have positive effects for both the economy and the 
environment. 

e Reusing, recycling, and using fewer resources can reduce environmental impacts. 

e Resources such as oceans, fresh water, and air, which are shared by everyone, need to be 
protected by careful planning and regulation of technological systems. 


e Some technologies can reduce the negative impacts of other technologies. 
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e Sustainable solutions are those that meet the needs of the present without compromising 


the ability of future generations to meet their own needs (Brundtland Commission, 1987). 


At the fourth-grade level, students are expected to know that sometimes technology can cause 
environmental harm. For example, litter from food packages and plastic forks and spoons discarded 
on city streets can travel through storm drains to rivers and oceans, where they can harm or kill 
wildlife. However, such negative effects can be lessened by reusing or recycling products as well as by 


reducing the amount of resources used in producing the products. 


Eighth graders are expected to recognize that technology and engineering decisions often 
involve weighing competing priorities, and there are no perfect solutions. For example, dams built to 
control floods and produce electricity have left wilderness areas under water and affected the ability 
of certain fish to spawn. They should be able to analyze such conflicts and be able to recommend 
changes that would reduce environmental impacts. For example, students could study the trade-offs 
involved in using paper or plastic to carry groceries or research the causes and effects of acid rain on 
forests and the costs of reducing those effects. They should know that designers can reduce waste 
by taking the entire life cycle of a product into account during design. For example, students should 
be able to discuss what a community could do when its landfill is close to capacity or find ways that 


designers of new products could reduce waste by considering the life cycle of a product. 


By 12th grade, students should have had a variety of experiences in which technologies were 
used to reduce the environmental impacts of other technologies, such as the use of environmental 
monitoring equipment. Students should be able to analyze the effects of different technologies on 
the environment—for example, by using data on the environmental impacts of power plants that use 
different types of fuel to inform decisions on which types of new power plants to build. Students 
should also be able to analyze complex human activities, such as energy generation, and propose sus- 
tainable solutions. Students could, for example, research the environmental impacts of energy gener- 


ation and create a presentation to a United Nations council on the trade-offs of various solutions. 
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Table 2.3 Effects of Technology on the Natural World assessment targets 


for grades 4, 8, and 12 


B. Effects of Technology on the Natural World 


Fourth graders should be aware that the use of technology can affect the environment, that the environment 
affects technology, and that reusing and recycling products can avoid damaging the environment. Eighth 
graders should be able to investigate the environmental effects of alternative decisions by tracing the life cycle of 
products and considering the trade-offs involved in different technologies. Twelfth graders should be aware that 
technologies used to monitor environmental change can help inform decision-making, and they should also be 
able to investigate complex global issues and generate innovative, sustainable solutions. 


Grade 4 


Students know that: 

T.4.5: The use of technology can 
affect the environment, including 
land, water, air, plants, and ani- 
mals. The environment also affects 
technology by providing sources of 
energy and raw materials. 


Grade 8 


Students know that: 

T.8.5: Some technological deci- 
sions involve trade-offs between 
environmental and economic 
needs, while others have positive 
effects for both the economy and 


environment. 


Grade 12 


Students know that: 

T.12.5: Many technologies have 
been designed to have a positive 
impact on the environment and 
to monitor environmental change 
over time to provide evidence for 


making informed decisions. 


T.4.6: Reusing and recycling mate- 
rials can save money while preserv- 
ing natural resources and avoiding 


damage to the environment. 


T.8.6: Resources such as oceans, 
fresh water, and air—which are 
essential for life and shared by 
everyone—are protected by regu- 
lating technologies in such areas as 
transportation, energy, and waste 


disposal. 


T.12.6: Development and modifi- 
cation of any technological system 
need to take into account how the 
operation of the system will affect 
natural resources and ecosystems. 


Students are able to: 

T.4.7: Identify the impact of a 
specific technology on the environ- 
ment and determine what can be 
done to reduce negative effects and 


increase positive effects. 


Students are able to: 

T.8.7: Compare the environmental 
effects of two alternative technolo- 
gies devised to solve the same prob- 
lem or accomplish the same goal 
and justify which choice is best, 
taking into account environmental 
impacts as well as other relevant 
factors. 


Students are able to: 

T.12.7: Identify a complex global 
environmental issue; develop a sys- 
tematic plan of investigation; and 
propose an innovative, sustainable 
solution. 
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C. Effects of Technology on the World of Information and Knowledge 


Human civilization owes its current form to a number of major revolutions in the capacity to 
communicate and pass along information. The genesis of writing, a technological development that 
began at least 3,000 years ago in ancient Mesopotamia, led to a flowering of commerce, mathemat- 
ics, science, and learning (Neugebauer, 1969; Van De Mieroop, 1999). Another milestone was the 
invention of the printing press by Johannes Gutenberg in the 15th century, which made it possible 
for ideas to be passed along to many people at widely distributed locations and times. Inventions of 
the telephone, telegraph, radio, movies, television, and the Internet all extended communication and 
expression. These revolutions changed the world of information and knowledge, resulting in trans- 


formative effects on society. 


Rapid advances in information and communication technologies during the latter half of the 
20th and early 21st centuries are creating an Information Age revolution. These technologies have 
made possible the storage, organization, and manipulation of vast quantities of data, far beyond 
what was possible for a physical library, and have greatly facilitated access to the information by any- 
one, anywhere. Together these technologies are modifying the world of information and knowledge 


itself, with implications for individuals, organizations, and entire societies. 


It is important for all citizens to understand the societal needs that led to the development of 
information and communication technologies and the effects, both positive and negative, of these 
technologies on the creation, extension, and use of knowledge and the expression of ideas. Students 
can acquire these insights and capabilities by taking courses in technology or simply through daily 
activities, such as studying the traditional subjects in school and pursuing personal interests at home. 
In other words, the process of using various technologies to access and interact with information 
and knowledge can yield valuable learning about the technology itself, and therefore contributes to 


students’ technology and engineering literacy. 


In contrast to the assessment targets later in this chapter under “Information and Communica- 
tion Technology,” which are concerned with student knowledge and the use of various technologies, 
the assessment targets in this subarea emphasize the revolutionary consequences of the Information 
Age—the ways in which society affects what information and knowledge is available and how the 


availability of that information profoundly affects society. 
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Key principles in the area of Effects of Technology on the World of Information and 


Knowledge that all students can be expected to understand at increasing levels of sophistication are: 


e Information technology is evolving rapidly, enabling ever-increasing amounts of infor- 
mation and data to be stored, managed, enhanced, analyzed, and accessed through a wide 
array of devices in various media formats. 

e Information and Communication Technology (ICT) enables the creation and modifica- 
tion of information and knowledge products by remotely connected individuals and teams. 

e The emergence of intelligent information technologies and the development of sophisti- 
cated modeling and simulation capabilities are transforming the world of information and 


knowledge, with potentially profound effects on society. 


Fourth-grade students should know that information technology provides access to vast 
amounts of information, that it can also be used to modify and display data, and that communica- 
tion technologies make it possible to communicate across great distances using writing, voice, and 
images. They should be able to identify examples of positive and negative impacts of these tools. 
For example, students should be able to identify positive effects of being able to send data to others. 
They should also be able to identify negative effects such as hasty email responses. 


Eighth graders should be aware of the rapid progress in development of ICT, should know how 
information technologies can be used to analyze, display, and communicate data, and should be able 
to collaborate with other students to develop and modify a knowledge product. For example, stu- 
dents should understand that translation tools on personal communication devices permit collab- 
oration with students from other countries on a school project, such as digital storytelling. Eighth 
graders should also understand that problems can be caused by using digital information or video 


without verifying quality. 


By 12th grade, students should have a full grasp of the types of data, expertise, and knowledge 
available online and should be aware of intelligent information technologies and the uses of simu- 
lation and modeling. They should also understand the potential disadvantage of uncritical use of 
the technologies. For example, they should know that false information can spread rapidly through 
the Internet and that these falsehoods, once established, can be difficult to correct. Students should 
be aware that the ubiquitous use of information communication and dissemination affects govern- 
ments, news, and other organizations as well as individuals, and they should understand the extent 


to which ICT has enabled a revolution in the world of knowledge. 
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Table 2.4 Effects of Technology on the World of Information and Knowledge 
assessment targets for grades 4, 8, and 12 


C. Effects of Technology on the World of Information and Knowledge 


Fourth graders should know that information, knowledge, and expertise can be accessed, collected, and shared 


by using information and communication technologies with positive and negative effects. Eighth-grade students 


should be able to critically weigh how information and communication technologies are rapidly evolving and 


changing the ways that people interact with information and each other. Twelfth graders should know that 


sophisticated information and communication technologies have transformed the world of information and 


knowledge itself, with positive and negative implications for society. 


Grade 4 


Students know that: 

T.4.8: Information technology 
provides access to vast stores of 
knowledge and information. This 
can result in positive and negative 
effects. 


T.4.9: Information technologies 
can be used to modify and display 
data in various ways that can be 


helpful or deceptive. 


Grade 8 


Students know that: 

T.8.8: Information technologies 
are developing rapidly so that the 
amount of data that can be stored 
and made widely accessible is grow- 
ing at a faster rate each year. 


T.8.9: Information technologies 
make it possible to analyze and 
interpret data—including text, 
images, and sound—in ways that 
are not possible with human senses 
alone. These uses may result in 


positive or negative impacts. 


Grade 12 


Students know that: 

T.12.8: Information technology 
allows access to vast quantities of 
data, expertise, and knowledge 
through a wide array of devices and 
formats to answer questions, solve 


problems, and inform the deci- 


sion-making process. 


T.12.9: Information technologies 
such as artificial intelligence, image 
enhancement and analysis, and 
sophisticated computer modeling 
and simulation create new types 
of information that may have 
profound effects on society. These 
new types of information must be 
evaluated carefully. 


T.4.10: Communications technol- 
ogies make it possible for people to 
communicate across large distances 


in writing, voice, and images. 


T.8.10: The large range of per- 
sonal and professional information 
technologies and communication 
devices allows for remote col- 
laboration and rapid sharing of 
ideas unrestricted by geographic 
location. 


T.12.10: The development of 
communication technologies that 
enable people to access vast quan- 
tities of information and publish 
their ideas globally has implications 
for governments, organizations, 
and individuals. 


Students are able to: 

T.4.11: Use information and com- 
munications technologies to access 
and interpret data and communi- 
cate with others. 


Students are able to: 

T.8.11: Use appropriate informa- 
tion and communication technol- 
ogies to collaborate with others on 
the creation and modification of 

a knowledge product that can be 
accessed and used by other people. 


Students are able to: 

T.12.11: Give examples to illus- 
trate the effects on society of the 
recording, distribution, and access 
to information and knowledge that 
have occurred in history, and dis- 
cuss the effects of those revolutions 
on societal change. 
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D. Ethics, Equity, and Responsibility 


Although technological advances have improved quality of life, newer technologies have 
sometimes resulted in negative effects, which in turn may have various ethical implications. Conse- 
quently, it is becoming increasingly important for citizens to recognize ethical issues related to the 
introduction and use of various technologies. For example, factories and power plants that benefit 
the citizens of one country may produce gases that cause acid rain, damaging forests in that country 
and neighboring countries. An ethical response to such a situation starts with the recognition that 


such effects are occurring, followed by concrete steps to mitigate the problem. 


The term “ethics” generally refers to a code of behavior or set of rules or guidelines for distin- 
guishing between right and wrong. In general, there is no such thing as a universally agreed-on set 
of ethical guidelines, since a behavior considered ethical in one culture or professional group may be 
considered unethical by a different group of people. Nonetheless, there are some ways of behaving 
and thinking that most people consider ethical, such as honesty and integrity (Resnik, 2007). The 
term “equity” generally refers to fairness, or equality of opportunity, while “responsibility” generally 


means holding oneself accountable to accomplish certain things, or to be trustworthy. 


While ethics, equity, and responsibility are important considerations in all human endeay- 
ors, they are especially important for the design, production, inspection, and use of technologies, 
as these have an immediate effect on people’s lives. Innumerable examples come to mind, such as 
the collapse of buildings during earthquakes, the failure of bridges due to poor maintenance, the 
distribution of poisonous foods due to unsafe processing methods, and lapses in the testing and 
correction of mechanical failures in automobiles. Prevention of such human tragedies involves dis- 
cussion of ethical practices and the responsibilities of individuals from technicians to engineers and 


policymakers. 


One sector of the current technological infrastructure that is especially vulnerable to unethical 
behaviors is the telecommunications sector and, in particular, the Internet. Access to the Internet 
offers unprecedented opportunities as well as challenges for students, as they have opportunities not 
only to access information but also to contribute and publish their own information for anyone in 
the world to read. But to use these tools (and others yet to be developed) in a responsible manner, 
students need to understand fundamental rules of ethical behavior with regard to the exchange of 
information. They also need to know how to protect themselves and to take personal responsibility 


for doing so. 
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Key principles in the area of Ethics, Equity, and Responsibility that all students can be 


expected to understand at increasing levels of sophistication are: 


e Technology by itself is neither good nor bad, but its use may affect others. 

e Not everyone has access to the same technologies. 

e Differences in available technologies within the United States and in other countries have 
consequences for public health and prosperity. 

e People living in one area need to be aware of how their use of technology affects the lives of 
people in other areas. 

e Storing information digitally requires a heightened attention to remote security threats. 


e It is important for people to take responsibility for the appropriate use of technology. 


Fourth graders should recognize that tools and machines can be helpful or harmful. For exam- 
ple, cars are very helpful for going from one place to another quickly, but their use can lead to acci- 
dents in which people are seriously injured. Students should also recognize that technology can be 


used in ways that hurt others, such as when a false rumor is posted about someone online. 


Eighth graders should be able to recognize that the potential for misusing technologies always 
exists and that the possible consequences of such misuse must be taken into account when making 
decisions. They should have a grasp of the technological inequalities around the world—as illus- 
trated by the existence of countries where few people can afford refrigerators and few schools have 
computers—and students should understand the economic and cultural reasons for these inequali- 
ties. They should know how to reduce the negative impacts that their use of technology may have on 
people who live in other areas. For example, they might consider avoiding the use of products that 
have been produced in ecologically vulnerable areas, such as the Amazonian rainforest. They should 
also have a solid understanding of a range of unethical and criminal behaviors involving the use of 


Internet and communications technologies. 


Twelfth graders should be able to take into account intended and unintended consequences in 
making technological decisions. They should understand the worldwide inequalities in technology 
access and know that efforts to transplant a technology from one culture to another should not be 
undertaken without a consideration of the costs and benefits to the society receiving the technology. 
They should be able to analyze the ethical responsibilities of various people in government and com- 


mercial enterprises, and demonstrate prudent and ethical use of communications technologies. 
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Table 2.5 Ethics, Equity, and Responsibility assessment targets for grades 4, 8, and 12 


D. Ethics, Equity, and Responsibility 


complex, and consider the consequences of unethical uses of technology. 


Fourth graders should know that tools and machines used carelessly might harm others, take responsibility for the 
appropriate use of tools and machines, and recognize misuses of communications and other technologies. Eighth 
graders should recognize that the same technologies are not available to everyone, and should take responsibility to 
reduce the negative impacts of technologies and increase their positive impacts. Twelfth graders should be able to 


take into account different viewpoints, recognize that transferring technologies from one society to another can be 


Grade 4 


Grade 8 


Grade 12 


Students know that: 
1.4.12: When using tools and 
machines, the results can be helpful 


or harmful. 


Students know that: 

T.8.12: Technology by itself is 
neither good nor bad, but its use may 
affect others; therefore, decisions 
about products, processes, and 
systems must take possible conse- 


quences into account. 


Students know that: 

T.12.12: Decisions made about 

the use of a technology may have 
intended and unintended conse- 
quences, and these consequences may 
be different for different groups of 
people and may change over time. 
Decisions about the use of a technol- 
ogy should consider different points 


of view. 


T.4.13: The technologies that people 
have available for essential tasks 

such as farming, cooking, medicine, 
transportation, and communication 


are vastly different in different parts 


of the world. 


T.8.13: People who live in different 
parts of the world have different 
technological choices and oppor- 
tunities because of such factors as 
differences in economic resources, 


location, and cultural values. 


T.12.13: Disparities in the technol- 
ogies available to different groups of 
people have consequences for public 
health and prosperity, but deciding 

whether to introduce a new technol- 
ogy should consider local resources 

and the role of culture in acceptance 


of the new technology. 


Students are able to: 

T.4.14: Explain the benefits and safe 
use of a tool or machine by showing 
how it can and should be used as well 
as how it should not be used and the 
consequences that may result if it is 


used inappropriately. 


Students are able to: 

T.8.14: Explain that it is important 
for citizens to reduce the negative 
impacts and increase the positive 
impacts of their technologies on 
people in another area or on future 


generations. 


Students are able to: 

1.12.14: Analyze responsibilities of 
individuals and groups, ranging from 
citizens and entrepreneurs to polit- 
ical and government officials, with 
respect to a controversial technolog- 


ical issue. 


7.4.15: Demonstrate the ethical 

use of information technologies by 
recognizing the ways that someone 
might harm someone else through 
the misuse of communication tech- 
nologies, and the kinds of informa- 
tion that could lead to abuse if widely 
shared. 


T.8.15: Explain why it is unethical 
to infect or damage other people’s 
computers with viruses or “hack” into 
other computer systems to gather or 


change information. 


T.12.15: Demonstrate the respon- 
sible and ethical use of information 
and communication technologies 

by distinguishing between kinds of 
information that should and should 
not be publicly shared and describing 


the consequences of a poor decision. 
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Design and Systems 


We live in a global society that is increasingly dependent on 
technology. In the drive to satisfy human needs and wants, peo- 
ple have developed and improved ways to communicate, move 
people and goods, build structures, make products, enhance 
ideas, cure diseases, use energy, and provide nutritious and safe 


food and water, as well as numerous other innovations. Techno- 


logical development has resulted in a vast network of products 


Technology 


and systems—often called “the designed world”—that is con- 
and Society 


stantly changing. The study of engineering design and systems 
is the study of the world in which all humans live and which all 


students will shape by the decisions they make as workers, consumers, and citizens. 


Because students live in a complex technological world, they face decisions every day that 
involve technology. Some of these are simple choices, such as deciding whether to use paper, plastic, 
or reusable bags for groceries or choosing which form of entertainment to enjoy, and others are more 
far-reaching and complex, such as which type of job to choose or what sort of medical treatment to 
select. How well students are prepared to make those choices depends in part on their understand- 
ing of technology. Essential knowledge and skills in this area of technology and engineering literacy 
are divided into four subareas: 


A. Nature of Technology; 
B. Engineering Design; 
C. Systems Thinking; and 


D. Maintenance and Troubleshooting. 


Understanding the Nature of Technology requires that one take a broad view. Simply put, 
technology satisfies the basic human needs for food and water, protection from the elements, health, 
energy, improved transportation, better and cheaper products, and improved communication. Stu- 
dents are expected to understand that the laws of nature provide limits on the types of technologies 
that can be developed. No one can create a perpetual motion machine, for example, as machines 


always require more energy input than they provide as useful output. 


Students are also expected to distinguish between science, technology, and engineering, and to 
recognize that science enables improvements in technology, while technological improvements cre- 
ated by engineers often lead to advances in science. Students should also recognize that some prob- 
lems can be solved through behavioral rather than physical changes, for example, by encouraging the 


use of carpools to relieve traffic congestion rather than constructing additional highway lanes. 
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Engineering Design is an iterative and systematic approach to creating solutions to a broad vari- 
ety of problems to meet people’s needs and desires. The process of design includes defining problems 
in terms of criteria and constraints; researching and generating ideas; selecting between or among 
alternatives; making drawings, models, and prototypes; optimizing, testing, evaluating the design, 


and redesigning if needed; and, eventually, communicating the results. 


Systems Thinking concerns the capability to identify the components, goals, and processes of 
systems. It also entails an understanding of such systems principles as feedback and control and also 


the ability to use simulations or other tools to predict the behavior of systems. 


Maintenance and Troubleshooting are how most people encounter technology on a daily 
basis—by troubleshooting technologies that malfunction and by maintaining tools and systems so 
they do not break down. The better a person understands the way something works, the easier it is 


to maintain it and to track down problems when they arise. 


Each of the above subareas relates to one of the broad categories included in the ITEEA Stan- 
dards for Technological Literacy. A table illustrating these connections is presented in appendix F. 


A. Nature of Technology 


Two of every three people in the United States think “technology” means computers and the 
Internet (Rose & Dugger, 2002; Rose, Gallup, Dugger, & Starkweather, 2004). Some people con- 
ceptualize technology somewhat more broadly to include cellphones and other electronics. How- 
ever, technology includes every way in which people manipulate the natural environment to satisfy 
their needs and wants. Frozen foods, paper cups, and clothing are examples of technology, as are 
dams, motorcycles, windmills, water treatment plants, flu shots, and grandfather clocks. Technology 


includes all of the various devices and systems that people make to fulfill some function. 


In addition to understanding the scope of technology, students are expected to understand how 
technology evolved and why the pace of technological change is so much faster today than in the 
past. For much of human history, technological knowledge was held by small groups of individuals 
who did not share it, but rather passed it guardedly from one generation to the next, sometimes 
from parent to child or master to apprentice. Today, by contrast, know-how is disseminated much 
more freely through a wide variety of educational institutions, both physically and online. Engineers 
and designers improve existing technologies, invent new devices and systems, and make technolog- 
ical breakthroughs that can be widely communicated in a short period of time, resulting in changes 
that can revolutionize entire industries. This is part of the reason that the rate of technological devel- 


opment is increasing at an unprecedented speed. 
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Another part of the reason can be found in today’s rapid advances in science. In many cut- 
ting-edge fields, such as bioengineering and nanotechnology, scientists and engineers work hand in 
hand, and sometimes the roles of scientist and engineer are taken on by a single person. An example 
of science pushing technology can be found in the breakthroughs in genetics that have made possi- 
ble new crops with higher yields and greater resistance to disease. Examples of technology pushing 
science can be found in the way that engineers provide more precise instruments, better collabora- 


tion tools, and evermore powerful computers. 


Tools and materials have also improved over time. From hand tools and power tools to com- 
puter probes and simulations, tools extend human capabilities, allowing people to see further and 
in greater detail, accomplish tasks more efficiently, and accomplish things that might otherwise be 
impossible. At the same time, new ways are constantly being developed to process raw materials to 
create products with properties unlike any in nature—self-cleaning clothing and paint, nano-fiber 
clothing that sheds water and never wrinkles, and composite materials for airplanes that are lighter 


and stronger than metal alloys, to name only a few. 


Key principles in the area of Nature of Technology that all students can be expected to under- 


stand at increasing levels of sophistication are: 


e Technology is constrained by laws of nature, such as gravity. 

e Scientists are concerned with what exists in nature; engineers modify natural materials to 
meet human needs and wants. 

e Technological development involves creative thinking. 

e Technologies developed for one purpose are sometimes adapted to serve other purposes. 

e Science, technology, engineering, mathematics, and other disciplines are mutually 
supportive. 

e The pace of technological change has been increasing. 


e Tools help people do things efficiently, accurately, and safely. 


Fourth-grade students are expected to distinguish natural and human-made materials, be famil- 


iar with simple tools, and recognize the vast array of technologies around them. 


Eighth graders should know how technologies are created through invention and innovation, 
should recognize that sometimes a technology developed for one purpose is later adapted to other 
purposes, and should understand that technologies are constrained by natural laws. They should 
also know that other resources besides tools and materials—energy, people, capital, and time—are 


generally needed to solve problems and meet design challenges. 


Twelfth graders should have an in-depth understanding of the ways in which technology 


coevolves with science, mathematics, and other fields; should be able to apply the concept of 
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trade-offs to resolve competing values; and should be able to identify the most important resources 


needed to carry out a task. 


Table 2.6 Nature of Technology assessment targets for grades 4, 8, and 12 


A. Nature of Technology 


to allow people to accomplish challenging tasks. 


Fourth graders should know that technology involves tools, materials, and creative thinking used to meet human 
needs and wants. Eighth graders should know that technology advances through invention and innovation and 
requires a variety of resources. Twelfth graders should know how technology coevolves with science and other fields 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

D.4.1: Scientists ask questions about 
the world; engineers create and 
modify technologies to meet people’s 


needs and desires. 


Students know that: 

D.8.1: Science is the systematic 
investigation of the natural world. 
Technology is any modification of 
the environment to satisfy people’s 
needs and wants. Engineering is the 
process of creating or modifying 
technologies and is constrained by 
physical laws and cultural norms, and 


economic resources. 


Students know that: 

D.12.1: Advances in science have 
been applied by engineers to design 
new products, processes, and systems, 
while improvements in technology 


have enabled breakthroughs in scien- 


tific knowledge. 


D.4.2: The improvement of existing 
technologies and the development 
of new technologies involve creative 


thinking. 


D.8.2: Technology advances through 
the processes of innovation and 
invention. Sometimes a technology 
developed for one purpose is adapted 
to serve other purposes. 


D.12.2: Engineers use science, 
mathematics, and other disciplines to 
improve technology, while scientists 
use tools devised by engineers to 
advance knowledge in their disci- 
plines. This interaction has deepened 


over the past century. 


D.4.3: Tools are simple objects that 
help people do things better or more 
easily, such as the cutting, shaping, 
and combining of materials that 


occur when making clothing. 


D.8.3: Tools have been improved 
over time to do more difficult tasks 
and to do simple tasks more effi- 
ciently, accurately, or safely. Tools 
further the reach of hands, voices, 


memory, and the five human senses. 


D.12.3: The evolution of tools, 
materials, and processes has played an 
essential role in the development and 
advancement of civilization, from the 
establishment of cities and industrial 
societies to today’s global trade and 


commerce networks. 


Students are able to: 

D.4.4: Inspect materials with 
different properties and determine 
which is most suitable for a given 


application. 


Students are able to: 
D.8.4: Simulate tests of various 
materials to determine which would 


be best to use for a given application. 


Students are able to: 
D.12.4: Take into account trade-offs 
among several factors when selecting 


a material for a given application. 


D.4.5: Choose an appropriate tool 
for accomplishing a task. 


D.8.5: Redesign an existing tool to 


make it easier to accomplish a task. 


D.12.5: Design a new tool to accom- 
plish a task more efficiently. 
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B. Engineering Design 


Engineering design (sometimes called technological design) is an iterative, systematic process for 
solving problems that involves creativity, experience, and accumulated disciplinary knowledge. As 
used in this framework, engineering design is a broad term, including processes such as architectural 


design, manufacturing design, industrial design, and software design. 


Much like scientific inquiry, engineering design is a dynamic process, not a rigid method. As 
engineering and science are often confused, it is helpful to draw a distinction. Scientific inquiry 
begins with a question and proceeds to generate and test hypotheses until the question is answered. 
In contrast, engineering design begins with a problem and proceeds to generate and test solutions 
until a preferred solution or solutions are reached. Whereas science seeks to understand, engineering 


seeks to meet people’s needs. 


The engineering design process usually begins by stating a need or want as a clearly defined 
challenge in the form of a statement with criteria and constraints. For example, a group of engineers 
might be given the task of designing a cellphone with a particular set of features, of a particular size 
and weight, with a certain minimum battery life, and that is able to be manufactured at a particular 
cost. Criteria are characteristics of a successful solution, such as the desired function or a particu- 
lar level of efficiency. Constraints are limitations on the design, such as available funds, resources, 
or time. Together, the criteria and constraints are referred to as the requirements for a successful 


solution. 


Once the challenge is defined, the next steps are often to investigate relevant scientific and 
technical information and the way that similar challenges have been solved in the past, and then to 
generate various possible solutions. This generation of potential solutions is the most creative part 
of the design process and is often aided by sketching and discussion. Using a process of informed 
decision-making, the designer or design team compares different solutions to the requirements of 
the problem and either chooses the most promising solution or synthesizes several ideas into an even 
more promising potential solution. The next step is usually to try out the solution by constructing 
a model, prototype (first of its kind), or simulation and then testing it to see how well it meets the 
criteria and falls within the constraints. An additional characteristic of engineering design is that 


ideas are tested before investing too much time, money, or effort. 
A person does not have to be an engineer to employ an engineering design process. Children can 


use this process to create a new toy, teachers can use it to plan a semester of lessons, and anyone can 


use it to address a need or desire encountered in everyday life. 
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The result of an engineering design process is not always a product. In some cases the result may 
be a process (such as a chemical process for producing an improved paint), a system (such as an air- 
line control system or a railway schedule), or a computer program (such as a video game or software 


to forecast the weather or model financial markets). 


When designing, it is important to take into account the entire life cycle of the product or pro- 
cess, including maintenance, troubleshooting, potential failure modes, impacts on the environment, 
and effects on society. Designing usually concludes with a presentation to clients or other interested 


parties (often classmates) on the preferred solution. 


Optimization, which is sometimes part of designing, means finding the best possible solution 
when some criterion or constraint is identified as the most important and others are given less 
weight. For example, optimizing the design of a pen might mean designing for lowest cost, best 
ink flow, or best grip, but not all three. Optimizing the design of an airplane engine usually means 
maximizing safety. In some engineering disciplines, the entire engineering process is referred to as 


“optimization under constraint.” 


It bears emphasizing that engineering design is not a rigid method. Different instructional mate- 
rials define it differently, although most definitions specify a sequence of steps, ranging from 5 steps 
for elementary students to 8 or 10 steps for high school students. The steps need not be followed in 
order. An experienced engineer might skip ahead a step or two or go back one or two steps. Or after 
generating solutions, it may become clear that the problem was poorly defined, and so it is best to 
restart the process from the beginning. Regardless of these differences in definition and use, engi- 


neering design always begins with a problem and ends with one or more solutions. 


Key principles in the area of Engineering Design that all students can be expected to under- 


stand at increasing levels of sophistication are: 


e Engineering design is a systematic, creative, and iterative process for addressing challenges. 

e Designing includes identifying and stating the problem, need, or desire; generating ideas; 
evaluating ideas; selecting a solution; making and testing models or prototypes; redesign- 
ing; and communicating results. 

e Requirements for a design challenge include the criteria for success, or goals to be achieved, 
and the constraints or limits that cannot be violated in a solution. Types of criteria and 
constraints include materials, cost, safety, reliability, performance, maintenance, ease of use, 
aesthetic considerations, and policies. 

e There are several possible ways of addressing a design challenge. 


e Evaluation means determining how well a solution meets requirements. 
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e Optimization involves finding the best possible solution when some criterion or constraint 
is identified as the most important and other constraints are minimized. 

e Engineering design usually requires one to develop and manipulate representations and 
models (for example, prototypes, drawings, charts, and graphs). 


Fourth graders should know that engineering design is a purposeful method of solving problems 
and achieving results. They should be able to state a simple design challenge in their own words, test 


a solution, and communicate the findings with drawings and models. 


Eighth graders should be able to carry out a full engineering design process to solve a problem of 
moderate difficulty. They should be able to define the challenge in terms of criteria and constraints, 


research the problem, generate alternative solutions, build and test a model or prototype, redesign, 
and communicate the findings. 


Twelfth graders should be able to meet a complex challenge, weigh alternative solutions, and use 
the concept of trade-offs to balance competing values. They should be able to redesign so as to arrive 
at an optimal solution. 
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Table 2.7 Engineering Design assessment targets for grades 4, 8, and 12 


B. Engineering Design 


Fourth graders should start to answer the question “How are technologies created?” by confronting simple yet systematic 


design challenges. Eighth graders should be able to use a more elaborate engineering design process, including problem 


definition, the use of prototypes, testing and iteration, and trade-offs. Twelfth graders should have a deep understanding and 


a broad array of design skills, including optimization. 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

D.4.6: Engineering design is a system- 
atic and creative process for meeting 
challenges. Often there are several 
solutions to a design challenge. Each 
one might be better in some way than 
the others. For example, one solution 
might be safer, while another might 


cost less. 


Students know that: 

D.8.6: Engineering design is a system- 
atic, creative, and iterative process for 
meeting human needs and wants. It 
includes stating the challenge, gener- 
ating ideas, choosing the best solution, 
making and testing models and proto- 
types, and redesigning. Often there are 


several possible solutions. 


Students know that: 

D.12.6: Engineering design is a com- 
plicated process in which creative steps 
are embedded in content knowledge 
and research on the challenge. Deci- 
sions on trade-offs involve systematic 
comparisons of all costs and benefits, 
and final steps may involve redesigning 


for optimization. 


D.4.7: Requirements for a design 
include the desired features of a 
product or system as well as the limits 
placed on the design, such as which 


materials are available. 


D.8.7: Requirements for a design are 
made up of the criteria for success and 
the constraints, or limits, which may 
include time, money, and materials. 
Designing often involves making trade- 
offs between competing requirements 


and desired design features. 


D.12.7: Specifications involve criteria, 
which may be weighted in various 
ways, and constraints, which can 
include natural laws and available 
technologies. Evaluation is a process for 
determining how well a solution meets 


the requirements. 


Students are able to: 
D.4.8: Use a systematic process to 


design a solution to a simple problem. 


Students are able to: 

D.8.8: Carry out a design process to 
solve a moderately difficult problem 
by identifying criteria and constraints, 
determining how they will affect the 
solution, researching and generating 
ideas, and using trade-offs to choose 


between alternative solutions. 


Students are able to: 

D.12.8: Meet a sophisticated design 
challenge by identifying criteria and 
constraints, predicting how these will 
affect the solution, researching and 
generating ideas, and using trade-offs to 
balance competing values in selecting 


the best solution. 


D.4.9: Construct and test a simple 
model to determine if it meets the 


requirements of a problem. 


D.8.9: Construct and test a model 
and gather data to see if it meets the 


requirements of a problem. 


D.12.9: Construct and test several 
models to see if they meet the require- 
ments of a problem. Combine features 


to achieve the best solution. 


D.4.10: Communicate design ideas 


using drawings and models. 


D.8.10: Communicate the results 

of a design process and articulate the 
reasoning behind design decisions by 
using verbal and visual means. Identify 
the benefits of a design as well as the 


possible unintended consequences. 


D.12.10: Communicate the entire 
design process from problem definition 
to evaluation of the final design, taking 
into account relevant criteria and con- 
straints, including aesthetic and ethical 
considerations as well as purely logical 


decisions. 
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C. Systems Thinking 


A system is any collection of interacting parts that make up a whole. In a sense, all technologies 
can be thought of as systems. Furthermore, the ways in which objects are produced and used can 
also be thought of as systems, as technological objects all have a life cycle, being made from raw 
materials, used, and eventually being discarded, at which point they may be recycled or added to 
landfills. More broadly, there are many examples of systems that are not technological in origin or 


else are only partly technology: ecosystems, financial systems, political systems, and so on. 


Beyond understanding that systems exist, citizens who are literate in technology and engineer- 
ing should be comfortable with the broader skill of systems thinking—a set of cognitive tools that 
increases in sophistication and power over time. Systems thinking is the capability to investigate—or 
think about—a system using certain principles. It enables people to understand complicated situa- 
tions that involve many interactions. For example, consider these principles: systems include sub- 
systems; any given system is typically part of one or more larger systems; and systems interact with 


other systems. 


These principles are important in thinking about the nation’s transportation system, among 
other things. The transportation system consists of a vast network of roads and rails and millions of 
vehicles. It is dependent on a second system that extracts oil from wells halfway around the world 
and carries that oil in thousands of supertankers to huge refineries, and from there to distribution 
points and gas stations. The combustion of fuels produces carbon dioxide; these systems affect the 
global climate system as well. Citizens who understand the effects of different fuels on the environ- 
ment will be able to make decisions about what kind of automobile to purchase based on both these 


interconnected technological systems and the price of gas. 


Systems thinking can be applied equally well to understanding systems other than purely tech- 
nological ones—for example, to analyze the way that communication technologies influence society 
and in turn are influenced by society, to think about the interplay between energy technologies and 
global climate change, and to predict future conditions given current trends and changes people may 
choose to make. It is a cognitive tool that helps people analyze problems they encounter in various 
settings and to propose solutions or determine reasonable courses of action. Simply put, systems 
thinking helps people understand how things are put together, how they function, and how they 


connect with other parts of the world, and it assists people in making informed decisions. 
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Key principles in the area of Systems Thinking that all students can be expected to understand 
at increasing levels of sophistication are: 


e Technological systems have parts that work together to accomplish a goal. 

e Systems may include subsystems and may interact with other systems. Systems may also be 
embedded within larger systems. 

e Dynamic technological systems require energy with more complicated systems tending to 
require more energy and to be more vulnerable to error and failure. 

e Technological systems are designed for specific purposes. They incorporate various pro- 
cesses that transform inputs into outputs. Two important features of technological systems 
are feedback and control. 


e Various methods can be used to increase the reliability of technological systems. 


Fourth graders should know that a system is a collection of interacting parts that make up a 
whole, that systems require energy, and that systems can be either living or nonliving. They should 
also be able to look at a simple system, identify its various parts, and recognize the functions of these 


parts within the larger system. 


Eighth graders should be able to analyze a technological system in terms of goals, inputs, pro- 
cesses, outputs, feedback, and control. They should be aware that systems can interact with each 
other and be able to identify the subsystems and components of a device by using reverse engineer- 
ing—the process of analyzing a system to see how it works in order to design a different device that 
performs the same function. Eighth graders should also be able to trace the life cycle of a product 
from raw materials to eventual disposal. 


Twelfth graders should be aware that technological systems are the product of goal-directed 
designs and that the building blocks of any technology consist of systems that are embedded within 
larger technological, social, and environmental systems. They should also be aware that the stabil- 
ity of a system is influenced by all of its components, especially those in a feedback loop. (Negative 
feedback tends to stabilize systems while positive feedback leads to instability.) Students should be 


able to use various techniques to forecast what will happen if a component or process is changed. 
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Table 2.8 Systems Thinking assessment targets for grades 4, 8, and 12 


C. Systems Thinking 


Fourth graders should be able to identify systems, subsystems, components, and boundaries in their everyday world 


and construct simple systems designed to accomplish particular goals. Eighth graders should be able to describe 


goals, inputs, outputs, and processes of systems; use reverse engineering and life cycles to analyze systems in terms of 


feedback and the flow of energy; and modify and construct moderately complicated systems. Twelfth graders should 


understand that systems are embedded in larger systems, recognize factors that stabilize systems, use systems for fore- 


casting, and redesign complicated systems to improve reliability. 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

D.4.11: All technological systems 
require energy and have parts that 
work together to accomplish a goal. 


Students know that: 

D.8.11: Technological systems are 
designed to achieve goals. They 
incorporate various processes that 
transform inputs into outputs. They 
all use energy in some form. These 
processes may include feedback and 


control. 


Students know that: 

D.12.11: The stability of a system 
depends on all of its components 
and how they are connected, with 
more complicated systems tending to 
require more energy and to be more 
vulnerable to error and failure. Neg- 
ative feedback loops tend to increase 


the stability and efficiency of systems. 


D.4.12: Many systems have subsys- 
tems within them and are defined by 
boundaries. Many systems are parts 


of larger systems. 


D.8.12: Technological systems can 
interact with one another to perform 
more complicated functions and 


tasks than any individual system can 


do by itself. 


D.12.12: Technological systems are 
embedded within larger technologi- 
cal, social, natural, and environmen- 


tal systems. 


Students are able to: 
D.4.13: Given a product, identify its 
systems, subsystems, and components 


by taking it apart. 


Students are able to: 

D.8.13: Examine a product or 
process through reverse engineering 
by taking it apart step by step to 
identify its systems, subsystems, and 
components, describing their interac- 
tions, and tracing the flow of energy 
through the system. 


D.12.13: Examine a system to pre- 
dict how it will perform with a given 
set of inputs in a given situation and 
how performance will change if the 
components or interactions of the 


system are changed. 


D.4.14: Create a diagram of a 
machine that contains multiple 
subsystems. Label the subsystems to 


explain what each one does. 


D.8.14: Measure and compare 

the production efficiency of two 
machines, a simple machine and 
a complex machine, designed to 


accomplish the same goal. 


D.12.14: Redesign a complex 
machine by modifying or rearranging 
its subsystems in order to optimize its 


efficiency. 


D.4.15: Construct a simple sys- 
tem to accomplish a goal, based on 
knowledge of the function of individ- 


ual components. 


D.8.15: Construct and use a mod- 
erately complicated system, given a 
goal for the system and a collection 
of parts, including those that may or 


may not be useful in the system. 


D.12.15: Construct and test a manu- 
facturing system composed of several 
machines to accomplish a given goal. 

Redesign the system to optimize its 


efficiency. 


Students are able to: 
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D. Maintenance and Troubleshooting 


The statement that “anything that can go wrong will go wrong,’ known as Murphy’s Law, has 
been attributed to aerospace engineer Edward Murphy, who first used the expression (or some- 
thing like it) in 1949 to explain the failure of measurement equipment for a high-speed rocket sled 
(Spark, 2006). Murphy’s Law has come to characterize everyday life, not only for engineers but also 
for everyone in modern society. Today humans are surrounded by and dependent on complicated 
devices that seem to go wrong at critical times. It is not uncommon to experience more than one 
technological failure in a single day, whether it’s a car that fails to start, a cellphone without “bars, 


or something as simple as an eyeglass frame with a lens that repeatedly pops out. 


A person who is literate in technology and engineering is aware that all technological sys- 
tems fail at one time or another and is therefore equipped with a foundation of concepts and 
skills that can be applied either to correct failed devices and systems or to prevent the failure from 
occurring in the first place. The most important of these concepts and skills are maintenance and 


troubleshooting. 


In the 2018 NAEP Technology and Engineering Literacy Framework, the term “maintenance” 
has a very specific meaning: It refers to keeping technological devices and systems in good condition 
so as to extend their useful life and reduce the number of breakdowns. For example, maintenance 
can refer to the regular upkeep of technologies so that they are less likely to fail, such as periodically 
replacing the oil in a car engine, cleaning the lint filter of a clothes dryer, or running regular software 


updates on a computer operating system. 


Troubleshooting, by contrast, refers to a systematic method of dealing with failures once they 
have occurred. It is common to begin troubleshooting by ascertaining the nature of the problem. For 
example, in the case of a television that has failed, it is important to determine if some parts of the 
device are still working. Is the power light on? Is the sound missing, or the picture, or both? If the 
power light is not on, it may be unplugged. If that is not the problem, the next step may be to isolate 
the problem to one part of the system. For instance, the problem may not be the TV at all but rather 
a faulty DVD, which can be tested by inserting a different DVD. A third step might be to learn as 
much as possible about how the system functions, either from an owner’s manual or from someone 
who is familiar with such systems. Troubleshooting is not confined to mechanical and electronic 


systems. Artists, writers, and musicians also encounter problems that require troubleshooting. 


Perhaps the most distinctive feature of troubleshooting is coming up with a number of different 
ideas about what may have caused the failure and then using a logical method for narrowing down 
the possible causes with a series of either-or tests, sometimes called a “fault tree,’ until the source of 


the problem is discovered. 
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When designing technological systems it is important for engineers to consider maintenance 
costs, as people may wish to pay alittle more for a product that is less expensive to maintain. Sim- 
ilarly, it is important for engineers to anticipate ways in which complicated products and systems 
are likely to break down and to build into the design simple ways to troubleshoot and fix the most 
common causes of failure. Factors to consider may include maintenance costs, available technolo- 


gies, time until obsolescence, and environmental impacts. 


Key principles in the area of Maintenance and Troubleshooting that all students can be 


expected to understand at increasing levels of sophistication are: 


e Tools and machines must undergo regular maintenance to ensure their proper functioning. 
e Troubleshooting is a systematic approach to diagnosing a technological failure. 


e Taking into account the entire life cycle of a product is an important part of designing. 


At the fourth-grade level, students should know that it is important to care for tools and 
machines so they can be used when they are needed. For example, tools should not be left out in 
the rain, and electronic equipment should be handled with care. Students should also know that 
if something does not work as expected, it is possible to find out what the problem is in order to 
decide if the item should be replaced or determine, if possible, how to fix it. They should know that 
some items, such as ballpoint pens, are designed to be disposable, and they should be able to discuss 


the disposal or recycling of such items. 


Eighth graders should be familiar with the concept of maintenance and should understand that 
failure to maintain a device can lead to a malfunction. They should also be able to carry out trou- 
bleshooting, at least in simple situations. For example, they should be able to safely use tools and 
instruments to diagnose a problem in a device, and they should be able to consult manuals or talk 
to experienced individuals to learn how the device works. They should be able to test various ideas 
for fixing the device. And they should be able to analyze an item’s life cycle and discuss the impact of 
disposing of an item that has reached the end of its useful life. 


By 12th grade, students should know that many devices are designed to operate with high 
efficiency only if they are checked periodically and properly maintained. They should also have 
developed the capability to troubleshoot devices and systems, including those that they may have 
little experience with. Students at this level should also be able to think ahead and to identify and 
document new maintenance procedures so that a malfunction is less likely to occur again. They 
should be able to weigh the costs and benefits of maintaining an existing item versus disposing of it 
and obtaining a newer replacement, with particular attention paid to lessening the environmental 


impact of disposing of obsolescent or nonfunctioning products. 


51 | 


Table 2.9 Maintenance and Troubleshooting assessment targets for grades 4, 8, and 12 


D. Maintenance and Troubleshooting 


reduce future failures. 


Fourth graders should recognize that tools and machines need to be cared for and that devices that fail can 
be fixed or replaced. Eighth graders should know that tools and machines must be maintained and be able 
to use a troubleshooting process to diagnose problems in technological systems. Twelfth graders should 
understand the importance of maintenance, be able to analyze malfunctions, and be able to devise ways to 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

D.4.16: It is important to care for 
different tools and machines in 
appropriate ways so that they are 
available to be used when needed. 


Students are able to: 

D.4.17: Change one aspect of 

a machine or tool at a time to 
discover why it is not working. 
Retest after each change has been 
made. 


Students know that: 

D.8.16: Many different kinds of 
products must undergo regular 
maintenance, including lubri- 
cation and replacement of parts 
before they fail so as to ensure 
proper functioning. 


Students are able to: 

D.8.17: Diagnose a problem ina 
technological device using a log- 
ical process of troubleshooting. 
Develop and test various ideas for 
fixing it. 


Students know that: 

D.12.16: Products and structures 
of various kinds can be redesigned 
to eliminate frequent malfunc- 
tions and reduce the need for 
regular maintenance. 


Students are able to: 

D.12.17: Analyze a system 
malfunction using logical rea- 
soning (such as a fault tree) and 
appropriate diagnostic tools and 
instruments. Devise strategies and 
recommend tools for fixing the 
problem. 


D.4.18: Identify the cause of fail- 
ure in a simple system and suggest 
ways that failure could be avoided 
in the future. 


D.8.18: Modify a moderately 
complicated system so that it 

is less likely to fail. Predict the 
extent to which these modifica- 
tions will affect the productivity 
of the system. 


D.12.18: Analyze a complicated 
system to identify ways that it 
might fail in the future. Identify 
the most likely failure points and 
recommend safeguards to avoid 
future failures. 


D.4.19: Recognize that all prod- 
ucts have a life cycle, starting with 
raw materials and ending with 
disposal or recycling. 


D.8.19: Trace the life cycle of a 
repairable product from incep- 
tion to disposal or recycling in 
order to determine the product’s 
environmental impact. 


D.12.19: Taking into account 
costs and current trends in 
technology, identify how long a 
product should be maintained 
and repaired and how it might 
be redesigned to lessen negative 
environmental impacts. 
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Information and Communication Technology (ICT) 


The integration of Information and Communication Tech- 
nologies (ICT) into every sphere of contemporary life has had 
profound implications for how people learn in school, solve 
practical problems, and function in the workplace. Networked 
computing and communications technologies and media have 
become essential tools of practically every profession and trade, 
including those of lawyers, doctors, artists, historians, electri- 


cians, mechanics, and salespersons. These devices make it possi- ee 
an ociety 


ble to redistribute learning and work experiences over time and 


space. Tools employed in various professions and trades, such 

as word processors, spreadsheets, audio, video, and photo editing tools, models, visualizations, and 
mobile wireless devices, are, in turn, being put to work in the study of core school subjects. Students 
are able to connect, access, and communicate with the wider world in ways that were unimaginable 
just a few years ago and that are continually changing. Particularly relevant for this framework is the 
fact that virtually all efforts to improve or create new technologies involve the use of ICT tools. And 
for many years to come, such novel technologies, computer-based and otherwise, will continue to 


bring about new approaches to education, work, entertainment, and daily life. 


As the term is used in this framework, ICT includes a wide variety of technologies, including 
computers and software learning tools, networking systems and protocols, hand-held digital devices, 
digital cameras and camcorders, and other technologies, including those not yet developed, for 


accessing, managing, creating, and communicating information. 


Although ICT is just one among many different types of technologies, it has achieved a special 
prominence in technology and engineering literacy because familiarity and facility with ICT is 
essential in virtually every profession in modern society, and its importance is expected to grow over 
the coming decades. A wide variety of ICT tools are routinely used in schools, the workplace, and 
homes. Rapidly evolving learning tools such as computers, online media, telecommunications, and 
networked technologies are becoming powerful supports for communities of learning and practice. 
Moving far beyond traditional text-based communication methods, the common language of global 
information sources and communication has broadened to include vast collections of images, music, 
video, and other media. Computers, networks, telecommunications, and media support collabora- 
tion, expression, and dissemination ranging from data organization and analysis, research, scholar- 
ship, and the arts to peer interactions. Ever-shrinking computer chips are put to work in a collection 
of devices that seems to be growing exponentially and that, at present, includes cellphones, digital 


assistants, media players, and geographical information systems, among a host of other devices. 
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Students should be aware of these devices and know how and when to use them. They must have 
mastered a wide range of ICT tools in common use, and they must have the confidence and capabil- 
ity to learn to use new ICT tools as they become available. Although students are not expected to 
understand the inner workings of these devices, they should have enough of an understanding of the 
principles underlying them to appreciate the basics of how they work. Five subareas of ICT literacy 
have been identified for assessment: 


A. Construction and Exchange of Ideas and Solutions; 
B. Information Research; 

C. Investigation of Problems; 

D. Acknowledgment of Ideas and Information; and 


E. Selection and Use of Digital Tools. 


Each of the above subareas relates to one of the broad categories included in the National Edu- 
cational Technology Standards for Students (NETSYS), and standards and frameworks developed 
by the Partnership for 21st Century Skills, the American Association of School Librarians, and the 
International Technology and Engineering Educators Association. The link between these subareas 
and the NETSYS and the Framework for 21st Century Learning is outlined in appendix E. 


A. Construction and Exchange of Ideas and Solutions 


Year after year, information and communication technologies challenge people to think, learn, 
and work in ways that were unimaginable only a short time ago and, as a result, enhance commu- 
nication and collaboration among individuals, groups, and organizations. Several lines of research 
have shown that teams are more productive than individuals for generating solutions to many kinds 
of problems, provided team members are effective collaborators. The findings extend to teams that 
collaborate online as well as face to face (Hennessey and Amabile, 2010). For schools, this continu- 
ing evolution translates into an increasing need to provide students with opportunities to develop 


digital and media communication skills and to collaborate in nontraditional learning environments. 


Several recent sets of national standards and many state standards cite effective communication 
skills and the capability to work collaboratively as essential for student success in the 21st century. In 
addition to mastering a set of computer-based skills, students should be able to employ a variety of 
media and technologies in order to communicate ideas, interact with others, and present informa- 
tion to multiple audiences. To be effective collaborators, students should be able to negotiate team 
roles and resources, draw on the expertise and strengths of other team members and remote experts, 
monitor progress toward goals, and reflect on and refine team processes for achieving goals. Since 


the assessment will be administered only to individuals, tasks involving collaboration with others 
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and sending and receiving communications will need to provide virtual, computer-based renditions 


of remote collaborators. 


Key principles in the area of Construction and Exchange of Ideas and Solutions that all 


students can be expected to understand at increasing levels of sophistication are: 


e Communication and collaboration are affected (in terms of quantity, quality, and results) 
by the choice of digital tools used. 

e Digital tools offer many options for formal and nonformal expression in nearly every aca- 
demic and professional discipline. 


e Teams need people with a variety of skills. 


Fourth-grade students should understand what is expected from members working as part of a 
team and should realize that teams are better than individuals at solving many kinds of problems. 
Students should be able to gather information from various sources and share ideas with a specified 
audience. 


Eighth-grade students should know that communicating always involves understanding the 
audience—the people for whom the message is intended. They should also be able to use feedback 


from others, and provide constructive criticism. 
Twelfth-grade students are expected to have developed a number of effective strategies for 


collaborating with others and improving their teamwork. They should be able to synthesize informa- 


tion from different sources and communicate with multiple audiences. 
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Table 2.10 Construction and Exchange of Ideas and Solutions assessment targets for 


grades 4, 8, and 12 


A. Construction and Exchange of Ideas and Solutions 


Fourth-grade students should be able to collaborate and communicate by working with other members 

of a (virtual) team to make decisions and develop presentations using a variety of formats. Eighth-grade 
students should be able to take into account the perspectives of different audiences, use a variety of media 
to create effective messages, and modify presentations based on feedback (virtual). Twelfth-grade students 
should have developed strategies to be effective collaborators, should be able to take into account multiple 
viewpoints, and should be able to synthesize information from a variety of sources. 


Grade 4 


Students know that: 

1.4.1: People collaborating as a 
team can often produce a better 
product than people working 
alone. There are common dig- 
ital tools that can be used to 
facilitate virtual or face-to-face 
collaboration. 


Grade 8 


Students know that: 

1.8.1: Collaboration can take 
many forms. Pairs or teams of 
people can work together in the 
same space or at a distance, at the 
same time or at different times, 
and on creative projects or on 
technical tasks. Different commu- 
nications technologies are used to 
support these different forms of 
collaboration. 


Grade 12 


Students know that: 

1.12.1: Effective collaboration 
requires careful selection of team 
members, monitoring of progress, 
strategies for reaching agreement 
when there are opposing points of 
view, and iterative improvement 
of collaborative processes. Infor- 
mation and communication tech- 
nologies can be used to record 
and share different viewpoints 
and to collect and tabulate the 
views of groups of people. 


Students are able to: 

1.4.2: Utilize input from (virtual, 
that is, computer-generated) col- 

laborators and experts or sources 

in the decision-making process to 
design a product or presentation. 


Students are able to: 

1.8.2: Provide feedback to a (vir- 
tual) collaborator on a product or 
presentation, taking into account 
the other person’s goals and using 
constructive, rather than negative, 
criticism. 


Students are able to: 

1.12.2: Work through a sim- 
ulation of a collaborative pro- 
cess. Negotiate team roles and 
resources, draw upon the exper- 
tise and strengths of other team 
members and remote experts, 
monitor progress toward goals, 
and reflect on and refine team 


processes for achieving goals. 


1.4.3: Communicate informa- 
tion and ideas effectively to an 
audience in order to accomplish a 


specified purpose. 


1.8.3: Communicate informa- 
tion and ideas effectively using 

a variety of media, genres, and 
formats for multiple purposes and 
a variety of audiences. 


1.12.3: Synthesize input from 
multiple sources to communicate 
ideas to a variety of audiences 
using various media, genres, and 
formats. 
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B. Information Research 


Research and information capability is a central ICT skill. In using digital and networking tools 
to find relevant and useful information, students must first be able to formulate a set of questions 
that will guide them in their search, and they must also be capable of synthesizing data from multi- 
ple sources. Students must be able to formulate efficient search strategies and to evaluate the cred- 
ibility of information and data sources. They must extract and save information and data that they 
judge to be relevant to the question at hand. And they must be able to use multiple ICT tools to 


organize, synthesize, and display information and data. 


Key principles in the area of Information Research that all students can be expected to under- 


stand at increasing levels of sophistication are: 


e Increases in the quantity of information available through electronic means and the ease 
with which knowledge can be published have heightened the need for the verification of 
sources of expertise. 

e Information can be distorted, exaggerated, or otherwise misrepresented. 

e Important strategies for ensuring quality of information include 1) assessing the source of 
information and 2) using multiple sources to verify the information in question. 

e Search strategies and skills are important capabilities in performing effective information 


research. 


Fourth-grade students should be aware of a number of digital and network tools that can be 
used for finding information, and they should be able to use these tools to collect, organize, and 


display data in response to specific questions and to help solve problems. 


Eighth-grade students should be aware of digital and network tools and be able to use them efhi- 
ciently. They should be aware that some of the information they retrieve may be distorted, exagger- 
ated, or otherwise misrepresented, and they should be able to identify cases where the information 


is suspect. 
Twelfth-grade students should be able to use advanced search methods and select the best digital 


tools and resources for various purposes. They should also be able to evaluate information for timeli- 


ness and accuracy, and they should have developed strategies to check the credibility of sources. 
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Table 2.11 Information Research assessment targets for grades 4, 8, and 12 


B. Information Research 


Fourth-grade students can use digital and network tools to find information and identify sources that may 
be biased in some way. Eighth-grade students are able to use digital resources to find information and also 
to recognize when information may be distorted, exaggerated, or otherwise misrepresented. Twelfth-grade 
students can use advanced search methods and select the best digital tools and resources for various pur- 
poses, can evaluate information for timeliness and accuracy, and can check the credibility of sources. 


Grade 4 


Students know that: 

1.4.4: Digital and network tools 
and media resources are helpful 
for answering questions, but 
they can sometimes be biased or 
wrong. 


Grade 8 


Students know that: 

1.8.4: Increases in the quantity 
of information available through 
electronic means and the ease 

by which knowledge can be 
published have heightened the 
need to check sources for possi- 
ble distortion, exaggeration, or 
misrepresentation. 


Grade 12 


Students know that: 
1.12.4: Advanced search tech- 
niques can be used with digital 


and network tools and media 
resources to locate information 
and to check the credibility and 


expertise of sources. 


Students are able to: 

1.4.5: Use digital and network 
tools and media resources to 
collect, organize, and display data 
in order to answer questions and 
solve problems. 


Students are able to: 

1.8.5: Select and use appropriate 
digital and network tools and 
media resources to collect, orga- 
nize, analyze, and display support- 
ing data to answer questions and 
test hypotheses. 


Students are able to: 

1.12.5: Select digital and network 
tools and media resources to 
gather information and data on a 
practical task, and justify choices 
based on the tools’ efficiency and 
effectiveness for a given purpose. 


1.4.6: Search media and digital 
sources on a community issue 
and identify sources that may be 


biased. 


1.8.6: Search media and digital 
resources on a community or 
world issue and identify specific 
examples of distortion, exagger- 
ation, or misrepresentation of 
information. 


1.12.6: Search media and digital 
resources on a community or 
world issue and evaluate the time- 
liness and accuracy of the infor- 
mation as well as the credibility of 
the source. 
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C. Investigation of Problems 


In addition to helping students find information, digital tools are widely used in core school 
subjects to support students’ critical thinking, problem-solving, and decision-making. In language 
arts courses, for example, students use online graphic organizers, word processors, and media as they 
read, analyze, and draw conclusions about various texts. They launch discussions on wikis to stim- 
ulate a rich consideration of topics prior to class time and to give students who are less confident in 
face-to-face situations the opportunity to be major contributors. In social science courses, students 
use databases and spreadsheets to create tables and graphs as they analyze and compare population 
densities in different historical periods. In science and mathematics, students use spreadsheets, visu- 
alization and modeling tools, digital probeware, and presentation tools to gather and interpret data 


on science and health issues. 


This subarea is closely related to the previous one, Information Research, but the focus here is on 
problem-solving and critical thinking as opposed to searching for information. In practice, of course, 
the two sets of skills will often be applied at the same time, but for the purpose of assessment it is 


useful to keep them separate. 


Schools are society’s means of preparing students for the world, and so many of the ways that 
students use digital tools reflect the way that professionals use similar tools to solve various practical 
problems, such as environmental issues, political conflicts, and economic challenges. In these cases, 
digital tools may be used to present the challenge scenario; to guide students in formulating the 
requirements of the challenge to be addressed; and to allow students to ask and answer significant 
questions, to exchange views with other students, sometimes in other cities or countries, to collect 
and analyze data, and then to develop and test various solutions through simulations. Other uses of 


digital tools in schools involve practical applications designed to prepare students for the responsi- 


bilities of adulthood. 


Key principles in the area of Investigation of Problems that all students can be expected to 


understand at increasing levels of sophistication are: 


e Digital tools can be very helpful in generating ideas and solving problems in academic sub- 
jects as well as in researching practical problems. 
e Digital models can be used to create simulations and test solutions. 


e Digital tools can be used to conduct experiments and investigate practical problems. 


Fourth-grade students should be able to use a variety of information and communication tech- 
nologies to investigate a local or otherwise familiar issue and to generate, present, and advocate for 
possible solutions. They should also be able to use digital tools to test hypotheses in various subject 


areas and to build models of simple systems. 
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Eighth-grade students should be able to use digital tools to identify and research a global issue 
and to identify and compare different possible solutions. They should also be able to use digital 
tools in testing hypotheses of moderate complexity in various subject areas in which they gather, 
analyze, and display data and draw conclusions. They should also be able to explore authentic 
issues by building models and conducting simulations in which they vary certain quantities to test 


“what if” scenarios. 


Twelfth-grade students should be able to use digital tools to research global issues and to fully 
investigate the pros and cons of different approaches. They should be able to design and conduct 
complex investigations in various subject areas using a variety of digital tools to collect, analyze, and 
display information and be able to explain the rationale for the approaches they used in designing 
the investigation as well as the implications of the results. Twelfth-grade students should also be able 
to conduct simulations, draw conclusions based on the results, and critique the conclusions based on 


the adequacy of the model to represent the actual problem situation. 
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Table 2.12 Investigation of Problems assessment targets for grades 4, 8, and 12 


C. Investigation of Problems 


Fourth-grade students are able to use digital tools to investigate local issues, test hypotheses, and build 
models. Eighth-grade students are able to use digital tools to investigate alternative solutions to global 
issues, test moderately complex hypotheses, build models, and conduct simulations. Twelfth-grade students 
can conduct more sophisticated investigations and simulations as well as recognize their limitations. For all 
levels, the focus is on types of hardware and software rather than on use of particular hardware or software 


products. 


Grade 4 


Grade 8 


Grade 12 


Students are able to: 

1.4.7: Use digital tools and 
resources to identify and inves- 
tigate a local issue and generate 
possible solutions. 


Students are able to: 

1.8.7: Use digital tools to identify 
a global issue and investigate pos- 
sible solutions. Select and present 
the most promising sustainable 
solution. 


Students are able to: 

1.12.7: Use digital tools and 
resources to identify a compli- 
cated global issue and develop a 
systematic plan of investigation. 
Present findings in terms of pros 
and cons of two or more innova- 
tive sustainable solutions. 


1.4.8: Use digital tools to test 
simple hypotheses in various 
subject areas. 


1.8.8: Use digital tools to gather 
and display data in order to test 
hypotheses of moderate complex- 
ity in various subject areas. Draw 
and report conclusions consistent 
with observations. 


1.12.8: Use digital tools to col- 
lect, analyze, and display data in 
order to design and conduct com- 
plicated investigations in various 
subject areas. Explain rationale 
for the design and justify conclu- 
sions based on observed patterns 
in the data. 


1.4.9: Use digital models to 
describe how parts of a whole 
interact with each other ina 
model of a system. 


1.8.9: Use a digital model of 
a system to conduct a simula- 
tion. Explain how changes in 
the model result in different 


outcomes. 


1.12.9: Having conducted a simu- 
lation of a system using a digital 
model, draw conclusions about 
the system, or propose possible 
solutions to a problem or ways to 
reach a goal based on outcomes of 
the simulation. Critique the con- 
clusions based on the adequacy of 
the model. 
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D. Acknowledgment of Ideas and Information 


Digital citizenship is an essential element of technology and engineering literacy. As rapid 
technological advances have increased people’s capacity to access and share information anytime and 
anywhere around the globe, there is increasing concern about the misuse and abuse of information. 
Some of the ethical and legal concerns were described under Technology and Society and include 
worries about such issues as providing false information, invading people’s privacy, “hacking” into 
secure networks, and using ICT tools for industrial espionage. There is therefore a certain amount 
of overlap between that subarea and this one, but this subarea is focused specifically on an especially 
important category of ethical issues: the appropriate use of intellectual property in the context of 


digital media. 


For many students the first opportunity to learn about the ethical implications of intellectual 
property appears in discussions about classroom cheating, in which a student looks at someone else’s 
test paper and writes down answers and ideas as his or her own. At the highest levels of academia, 
this practice is known as “plagiarism,” and allegations of plagiarism can lead to criminal as well as 
ethical sanctions. On the other hand, it is not cheating to incorporate other people’s ideas as long as 
credit for the source of the ideas is given at the time they are used. It is therefore essential that stu- 
dents know the conventional methods for appropriately crediting others’ ideas, words, and images, 


both orally and in the form of writing and other media. 


A closely related issue is the use and misuse of copyrighted material. Even at the elementary level 
it has become so easy to copy and share digital information that children need to understand the 
importance of respecting copyrighted materials so that they will be more likely as adults to continue 
to honor intellectual property rights and laws protecting patents, trademarks, copyrights, music, and 
video. Although technological safeguards may be developed in future years, individual respect for 


the intellectual property of others will continue to be an important ethical imperative. 


Key principles in the area of Acknowledgment of Ideas and Information that all students 


can be expected to understand at increasing levels of sophistication are: 


e Copyright laws and policies are designed to protect intellectual property. 

e Fair use guidelines are designed to support the use of copyrighted materials for academic 
purposes and for journalism and other forms of writing and commentary. 

e There are multiple guiding principles (laws, policies, and guidelines) that govern the use of 


ideas and information. 


Fourth-grade students should understand that it is permissible to use others’ ideas as long as 


appropriate credit is given. This ethical guideline that one should give credit where it is due holds 
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true not just for tests and homework, but also even in everyday conversation. They should also know 


that copyrighted materials cannot be shared freely. 


Eighth-grade students should be aware of general principles concerning the use of other people’s 
ideas and know that these principles are the basis for such things as school rules and federal laws 
governing such use. They should know about the limits of fair use of verbatim quotes and how to 
cite sources in papers or other media productions. They should understand the importance of 
giving appropriate credit for others’ ideas and contributions, and they should know how to give 


such credit. 
Twelfth-grade students should understand the fundamental reasons for intellectual property 


laws and should know acceptable practices for citing sources when incorporating ideas, quotes, and 


images into their own work. 
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Table 2.13 Acknowledgment of Ideas and Information assessment targets for 


grades 4, 8, and 12 


D. Acknowledgment of Ideas and Information 


from others. 


Fourth-grade students exhibit digital citizenship by understanding that it is permissible to use others’ ideas 
as long as appropriate credit is given but that copyrighted materials cannot be shared freely. Eighth-grade 
students should be aware of and comply with laws and ethical guidelines for incorporating ideas, text, and 
images into their own work. Twelfth-grade students should understand the reasons for protecting intel- 
lectual property and demonstrate responsible and ethical behaviors when using ideas, quotes, and images 


Grade 4 


Students know that: 

1.4.10: It is allowable to use 
other people's ideas in one’s own 
work provided that proper credit 
is given to the original source, 
whether information is shared in 


person or through ICT media. 


Grade 8 


Students know that: 

1.8.10: Style guides provide 
detailed examples for how to give 
appropriate credit to others when 
incorporating their ideas, text, or 
images in one’s own work. 


Grade 12 


Students know that: 


1.12.10: Legal requirements 
governing the use of copyrighted 
information and ethical guide- 
lines for appropriate citations are 
intended to protect intellectual 


prop erty. 


Students are able to: 

1.4.11: Identify or provide 
examples demonstrating respect 
for copyrighted material, such as 
resisting the request from a friend 
to copy a song from a CD or plac- 
ing copyrighted material online. 


Students are able to: 

1.8.11: Identify or provide exam- 
ples of fair use practices that apply 
appropriate citation of sources 
when using information from 
books or digital resources. 


Students are able to: 

1.12.11: Identify or provide 
examples of responsible and ethi- 
cal behavior that follow the letter 
and spirit of current laws concern- 
ing personal and commercial uses 
of copyrighted material as well as 
accepted ethical practices when 
using verbatim quotes, images, or 
ideas generated by others. 
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E. Selection and Use of Digital Tools 


Until recently, classroom uses of technology tended to focus almost exclusively on helping stu- 
dents become competent users of the technology itself: Educators today have come to recognize that 
how technology is used as a tool for learning is at least as important—if not more so—than simply 
how to use technology. Nonetheless, students still require basic operational skills and concepts to 
be effective users of technology for learning. These skills include the capabilities to select and use 
the appropriate tools, to use those tools to complete tasks effectively and productively, and to apply 


current knowledge about technology to learn how to use new technologies as they become available. 


Key principles in the area of Selection and Use of Digital Tools that all students can be 


expected to understand at increasing levels of sophistication are: 


e Knowledge about the common uses of readily available digital tools supports effective tool 
selection. 
e A fundamental aspect of technology and engineering literacy is the possession of founda- 


tional ICT skills in the use of common productivity tools. 


Fourth-grade students should know that different digital tools have different purposes. They 
should also be able to use a variety of digital tools that are appropriate for their age level. For exam- 
ple, they should be reasonably competent in using digital tools for creating documents and images, 
for solving problems, and for collecting and organizing information. 


Eighth-grade students should be familiar with different types of digital tools and be able to 
move easily from one type of tool to another—for example, creating a document or image with one 
tool and then using a second tool to communicate the result to someone at a distant location. They 


should be able to select and use effectively a number of tools for different purposes. 
Twelfth-grade students should be competent in the use of a broad variety of digital tools and be 


able to explain why some tools are more effective than others that were designed to serve the same 
purpose, based on the features of the individual tools. 
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Table 2.14 Selection and Use of Digital Tools assessment targets for grades 4, 8, and 12 


E. Selection and Use of Digital Tools 


task, by referencing specific features. 


Fourth-grade students know that different digital tools have different purposes and are able to use a number of 
different tools. Eighth-grade students can categorize digital tools by function and can select appropriate tools and 
demonstrate effective use of the tools for different purposes. Twelfth-grade students are competent in the use of a 


broad variety of digital tools and can justify why certain tools are chosen over others that might accomplish the same 


nizing, analyzing, and presenting 
information, while other kinds of 
tools are appropriate for creating text, 
visualizations, and models and for 


communicating with others. 


Grade 4 Grade 8 Grade 12 
Students know that: Students know that: Students know that: 
1.4.12: Different digital tools have 1.8.12: Certain digital tools are 1.12.12: A variety of digital tools 
different purposes. appropriate for gathering, orga- exist for a given purpose. The tools 


differ in features, capacities, oper- 
ating modes, and style. Knowledge 
about many different ICT tools is 
helpful in selecting the best tool for a 


given task. 


Students are able to: 

1.4.13: Use digital tools (appropriate 
for fourth-grade students) effec- 
tively for different purposes, such as 


searching, organizing, and presenting 


Students are able to: 

1.8.13: Use appropriate digital tools 
to accomplish a variety of tasks, 
including gathering, analyzing, and 


presenting information as well as cre- 


Students are able to: 

1.12.13: Demonstrate the capability 
to use a variety of digital tools to 
accomplish a task or develop a solu- 


tion for a practical problem. Justify 


information. ating text, visualizations, and models | the choice of tools, explain why other 
and communicating with others. tools were not used based on specific 
features of the tools, and summarize 
the results. 
Conclusion 


This chapter has described in detail the knowledge and capabilities related to technology that 


will be assessed on the 2018 NAEP Technology and Engineering Literacy Assessment. While these 
assessment targets represent an important component of technology and engineering literacy, they 
are not intended to limit the scope of concepts and capabilities that could be addressed in curricula. 
As NAEP is an on-demand test, important knowledge and capabilities that should form a part of 


every students’ education but are difficult to assess have been excluded from this framework. 


Although chapter two is central to the design of the NAEP Technology and Engineering Lit- 
eracy Assessment, it is not sufficient in itself to describe the kinds of reasoning to be expected from 
students, the context or subject matter that will be used to construct test items, or the overall shape 


of the entire assessment. These issues are taken up in chapters three and four. 
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CHAPTER THREE 


PRACTICES AND CONTEXTS FOR TECHNOLOGY 
AND ENGINEERING LITERACY 


Introduction 


Chapter two described the assessment targets for technology and engineering literacy, both 
knowledge and skills that NAEP will assess. This chapter describes three generalizable practices that 
represent the kinds of thinking and application that will be expected of students across the three 
major assessment areas. The chapter also explains the contexts, or situations and types of problems, 
in which assessment tasks and items will be set. These three elements and their relationships are 


portrayed in figure 2. 


PRACTICE 


Understanding Technological 
Principles 
Developing Solutions and 


CONTEXT 
Societal Issues 
Design Goals 


Achieving Goals Schools and 


Communicating and Community Problems 
Collaborating Etc. 


ASSESSMENT 
AREAS 
Technology and 
Society 
Design and Systems 
Information and 
Communication 
Technology (ICT) 


Figure 2 — Elements of the NAEP Technology and Engineering Literacy Assessment 
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As indicated in figure 2, the practices expected of students are general, crosscutting reasoning 
processes that students must use to show that they understand and can use their technological 
knowledge and skills. The contexts in which technology and engineering literacy tasks and items 
appear will include typical issues, problems, and goals that students might encounter in school or 
practical situations. The particular knowledge and skills that are the targets of the assessment lie in 
the three assessment areas. Together, the assessment targets, practices, and contexts provide a struc- 
ture for the generation of tasks and items. This chapter describes the practices and contexts in some 
detail and provides examples of the types of tasks and items that result when these three elements 
are combined. 


Practices 


Practices contribute to the framework by articulating the general kinds of thinking and reason- 
ing that students are expected to demonstrate when responding to assessment tasks and items. The 
framework specifies three generic kinds of practices that apply across the three assessment areas: 
(1) understanding technological principles; (2) developing solutions and achieving goals; and (3) 
communicating and collaborating. 


Building on the pyramid of major assessment 
areas introduced in chapter two, the practices are 
distributed around a circle in which the pyramid 
sits. As critical components of technological liter- 
acy, these crosscutting practices are applied across 
all three major assessment areas. For example, 
communicating effectively and collaborating with 
others are necessary skills for understanding the 
effects of technology on the natural world, design- 
ing an engineering solution to a technological 
problem, and achieving a goal using information 


and communication technologies. 


The Science Framework for the 2009 National Assessment of Educational Progress served as the 
primary source for these practices. Developers of the science framework had examined the sections 
on Science and Technology and the Designed World from the National Science Education Standards 
as well as Benchmarks for Scientific Literacy and cognitive research on science learning. Learning 
research has shown that these three kinds of cognitive processes represent how all individuals build 
their content knowledge and understanding in a subject area and how they develop strategies for 
using their knowledge in their thinking, reasoning, and application to new situations. To create the 


practices for the NAEP Technology and Engineering Literacy Framework, the crosscutting practices 
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from the Science Framework for the 2009 NAEP were modified so that they applied to processes 
relevant to technological literacy, primarily by indicating that technology and engineering literacy 
depends on having and using knowledge about technologies to reason, develop solutions, communi- 
cate, and collaborate. In addition, the national, international, and state technology and engineering 


literacy frameworks cited in chapter two were used as reference points. 


Although the practices are related and not independent of each other, classifying the assessment 
targets in chapter two according to the three crosscutting practices will help developers produce a 
range of rich and challenging assessment tasks and items. A brief description of each of these three 


practices is offered below. 


Understanding Technological Principles focuses on students’ knowledge and understanding 
of technology and their capability to think and reason with that knowledge. This practice ranges 
from the knowledge of simple declarative facts and concepts to higher-level reasoning about facts, 
concepts, and principles and their interrelationships. Students should be able to call on their rec- 
ognition and understanding of technological principles to explain features and functions of tech- 
nologies and systems, how components fit together, and to make predictions, comparisons, and 
evaluations. The assessment targets that elicit this practice require students to identify examples, 
explain, describe, analyze, compare, relate, and represent the technological principles specified in 
chapter two. This practice also includes understanding the relationships among components of 


systems and interactive processes. 


Developing Solutions and Achieving Goals refers to students’ systematic application of tech- 
nological knowledge, tools, and skills to address problems and achieve goals presented in societal, 
design, curriculum, and realistic contexts. This practice includes both procedural and strategic capa- 
bilities—knowing how to apply simple steps and use technological tools to address authentic tasks, 
as well as when and where to apply the tools and design and problem-solving strategies. This practice 
draws on the previous practice—to understand technological principles—and adds the dimension 
of applying this knowledge to solve problems and achieve goals. This practice involves using fun- 
damental problem-solving processes such as planning, monitoring, evaluating, and revising, and 
how these generic problem-solving strategies can be employed in the three assessment areas. It may 
engage students in analyzing goals, planning, designing, and implementing as well as in iteratively 
revising and evaluating possible solutions to meet the requirements of a problem or to achieve a 
goal. For the NAEP Technology and Engineering Literacy Assessment, a distinguishing feature 
of this practice is that the students respond to questions and tasks during the process of solving a 
multistage problem or working through how best to achieve a goal, using their understanding of 


technological principles to do so. 
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Communicating and Collaborating centers on students’ capabilities to use contemporary 
technologies to communicate for a variety of purposes and in a variety of ways, working individually 
or in teams. In the three major assessment areas, in order to address societal issues, solve problems, 
achieve goals, and design processes and products, students must develop representations and share 
ideas, designs, data, explanations, models, arguments, and presentations. Effective teamwork and 
collaboration with peers and experts who are either present or in another location are important 
skills that can help students achieve their goals. In the assessment, collaboration tasks will engage 
individual students with virtual (computer-generated) peers and experts. Communication and col- 
laboration are critical, crosscutting practices in all subject domains. For technology and engineering 
literacy, these practices are distinguished by students’ facility with a range of technologies to com- 


municate and collaborate. 


Table 3.1 presents generic examples of how these three practices can be used to classify targets 
in the three major assessment areas. These are sample ideas for items and tasks and will not be used 
in the actual assessment. It should again be noted that the boundaries between the practices are not 
entirely distinct, but referring to these three practices can be helpful in the development of tasks and 


items and the interpretation of student performance for a range of cognitive demands. 
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Table 3.1 Classification of types of assessment targets in the three major assessment 
areas according to the practices for technology and engineering literacy 


Technology and Society 


Design and Systems 


Information and 
Communication 


Technology (ICT) 


Analyze advantages and disad- 
vantages of an existing technology 
Explain costs and benefits 
Compare effects of two technolo- 


gies on individuals 


Describe features of a system or 
process 
Identify examples of a system or 
process 


Explain the properties of dif- 


Describe features and functions 
of ICT tools 

Explain how parts of a whole 
interact 


Analyze and compare relevant 


Understanding Propose solutions and ferent materials that determine features 
Technological alternatives which is suitable to use fora given | Critique a process or outcome 
Principles Predict consequences of a application or product Evaluate examples of effective 
technology Analyze a need resolution of opposing points of 
Select among alternatives Classify the elements ofa system | view 
Justify tool choice for a given 
purpose 
Select appropriate technology to | Design and Build a product Select and Use appropriate tools 
solve a societal problem using appropriate processes and to achieve a goal 
Develop a plan to investigate an | materials Search media and digital 
issue Develop forecasting techniques resources 
Gather and Organize data and Construct and Test a model or Evaluate credibility and solutions 
information prototype Propose and Implement 
Developing Analyze and Compare advan- Produce an alternative design or | strategies 


Solutions and 


tages and disadvantages of a 


product 


Predict outcomes of a proposed 


Achieving Goals | proposed solution Evaluate trade-offs approach 
Investigate environmental and Determine how to meet a need Plan research and presentations 
economic impacts ofa proposed _| by choosing resources required to | Organize data and information 
solution meet or satisfy that need Transform from one representa- 
Evaluate trade-offs and impacts Plan for durability tional form to another 
of a proposed solution Troubleshoot malfunctions Conduct experiments using 
digital tools and simulations 
Present innovative, sustainable Display design ideas using models | Plan delegation of tasks among 
solutions and blueprints team members 
Represent alternative analyses Use a variety of media and Provide and Integrate feedback 
and solutions formats to communicate data, from virtual peers and experts to 
Display positive and negative information, and ideas make changes in a presentation 
consequences using data and Exhibit design of a prototype Critique presentations 
Lem media Represent data in graphs, tables, Express historical issues in a 
Communicating . . . . ; 
: Compose a multimedia and models multimedia presentation 
and Collaborating 


presentation 

Produce an accurate timeline of a 
technological development 
Delegate team assignments 
Exchange data and information 


with virtual peers and experts 


Organize, Monitor, and Eval- 
uate the effectiveness of design 
teams 

Request input from virtual 
experts and peers 


Provide and Integrate feedback 


Argue from an opposing point 
of view 

Explain to a specified audience 
how something works 

Address multiple audiences 
Synthesize data and points of 


view 
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Examples of Practices Applied in Each of the Assessment Areas 


The following sections describe how the three practices of Understanding Technological Prin- 
ciples, Developing Solutions and Achieving Goals, and Communicating and Collaborating can be 
used to classify the general types of thinking and reasoning intended by the assessment targets in the 
three major assessment areas of Technology and Society, Design and Systems, and Information and 


Communication Technology (ICT). 


Practices Applied in Technology and Society 


Assessment targets in the area of Technology and Society are concerned with the effects of tech- 
nology on human society, the natural world, and the world of information and knowledge as well as 


with issues of ethics, equity, and responsibility. 


Understanding Technological Principles 


To provide evidence that they understand principles in these three subareas, students could 
be asked to perform a variety of tasks, such as recognizing examples of the effects of technologies; 
identifying examples of ethical and equity issues; describing local and global effects of technologies; 
explaining the effects of rapidly changing technologies on knowledge creation, access, and manage- 
ment; analyzing beneficial and negative impacts; recognizing examples of responsible, ethical uses of 
technologies; comparing costs and benefits of technologies; predicting potential impacts on society 


and the environment; and explaining the relationships among technologies. 


Developing Solutions and Achieving Goals 


Students must use their understanding of the technological principles for Technology and Soci- 
ety specified in chapter two to apply that knowledge as they address novel issues and problems. To 
demonstrate their capacity to address issues and problems in the assessment area of Technology and 
Society, students could be asked, for example, to develop alternative proposals for a new technology 
based on an analysis of potential positive and negative impacts. Problem-solving practices could be 
demonstrated in a series of tasks and items involved in analyzing the uses of the new technology, 
gathering data and information on its impacts, analyzing the data, interpreting results, and evaluat- 


ing alternatives. 


Communicating and Collaborating 


To communicate and collaborate with others (virtual others in the assessment) in the course to 
respond to issues, students must draw on their understanding of the technological principles spec- 
ified in chapter two and apply their knowledge as they work through given problems and issues. 


For example, to address issues in Technology and Society, students can use a variety of modalities 
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to represent and exchange data, ideas, and arguments about the advantages and disadvantages of 
technologies. Students can collaborate (virtually) to form teams that will gather and integrate infor- 
mation about the potential impacts of a technology on human society or the natural world. Stu- 
dents can evaluate the qualifications, credibility, and objectivity of virtual experts. Tasks can require 
students to demonstrate their capability to interact, collaborate, and contribute to work as a team. 


Students can use various media and representations to share their analyses and recommendations. 


Table 3.2 provides some examples of how the three practices can be applied to assessment targets 
for Technology and Society to generate tasks and items at the middle school level. The key princi- 
ples and targets were selected from tables 2.2-2.5 in chapter two. These are sample ideas for items 
and tasks and will not be used in the actual assessment. Simpler tasks could be developed for grade 4 


targets and more complex tasks could be developed for grade 12 targets. 
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Table 3.2 Examples of grade 8 tasks representing practices in each subarea of 
Technology and Society 


A. Interaction of B. Effects of C. Impacts on D. Ethics, Equity, 
Technology and | Technology on the the World of and Responsibility 
Humans Natural World Information and 
Knowledge 
Selected | The relationship Reusing, recycling, | Information tech- | Technology by 
Principles | between technol- and using fewer nology is evolving | itself is neither 
ogy and society is _| resources can reduce | rapidly, enabling good nor bad, but 
reciprocal. Society | environmental ever-increasing its use may affect 
drives technolog- impacts. amounts ofinfor- | others. 
ical change, while mation and data to 
changing technol- be stored, managed, 
ogies in turn shape enhanced, ana- 
society. lyzed, and accessed 
through a wide array 
Practices of devices in various 
media formats. 
Explain what factors | Identify and Compare the What are the pos- 
need to go into a provide a rationale | impact of geo- itive and negative 
Und di decision to change _| for appropriate graphical informa- | consequences of the 
ee : Bel the use of a river and inappropriate tion systems and predicted change 
Technological ; . 7 - : f ; dici- 
Painciples and identify possi procedures for dis. 14th century maps rom print to digi 
ble consequences of | posing of electronic | on people’s capabil- | tal news? 
doing so. devices. ity to explore new 
territory. 
The community has | Given a specific Use a simulation to | What processes and 
Developing decided to imple- consumer electron- | test the adequacy digital tools might 
Ssiaeeiend ment a new wind ics product suchas | of exit routes for the city council put 
Achieving turbine system. a cellular telephone, | evacuating residents into place in order 
Goals Design an investiga- | design a new way to | of amountain town | to make sure all citi- 
tion into the impact | increase its appro- during a wildfire. zens have a say? 
onthe community. | priate disposal. 
Collaborate with Organize a cam- Present a set of Debate with 
engineers and urban | paign witha virtual | images from two a virtual team 
planners (virtual) team to inform artists of the period | member the privacy 
Communicating | through a website to | the public of the that represent differ- | and safety issues 
and collect and commu- | dangers of improper | ent perspectives on | involved in estab- 
Collaborating | nicate data about disposal of con- a major event. lishing interna- 
the effects of a wind | sumer electronic tional projects. 
turbine system on | products. 
the community. 
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Illustrative Tasks and Items 


The following examples represent three types of tasks and items that could be used to assess 
targets related to the Technology and Society assessment area. The first example is a scenario-based 
set of items, and the next two are familiar, conventional constructed and selected-response items. 
These examples were used to guide development of the first NAEP Technology and Engineering 
Literacy Assessment, administered in 2014. Released tasks from the 2014 assessment are available in 
the 2014 NAEP Technology and Engineering Literacy Report Card. 


Scenario-Based Item Sets 


Items of this type present students with a problem or goal set within a broader context. The 
example below calls on the student to employ the practice of Developing Solutions and Achieving 
Goals. In this task, the student must use ICT tools to analyze the impacts of technology on human 
society. This example employs a computer-based interactive format in which students search for 
information concerning video games and violence and then use word-processing, spreadsheet, and 
Web-based tools to develop a PowerPoint presentation. The PowerPoint presentation calls on the 


crosscutting practice of Communicating and Collaborating. 


Context: Human Society 
Topic: Video Games and Violence 
Target Level: Grade 12 


TASK DETAILS 


You are going to create a slide show about the relationship between Video 
Games and Violence to present to your class. 


You should use no more than five slides. 
Your slide show will be assessed on: 


® the ideas and information you include; 
« the way you organise the information; 
e the design of the presentation; and 

e your use of the software features. 


This is the big task for this module. 

After this big task you will have one small task to complete but this should 
only take one minute. 
You should spend most of your remaining time on this big task. 


Before you begin this task you will watch a demonstration of how to do it. 


Time remaining Click on ‘I’ve finished’ to watch the demonstration. 
24 = minute(s) 


* 
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File Edit Format Tools Slide Show Help 
DSEBA BA FH 4BBP/9-+- \MOBeEaee -/O8 
5 Arial +18 +)/B Zz u l[E = f= | A) = =| A~| wren new side & Bia@ 


CPCI ee i a Ce) 


‘Click to add notes 


Bs «| » 
Time remaining : Autoshapesy \ WOOK A Bd) o-4-A-= 
23 minutes = Dara 


‘Create a slide show to present your argument. Use up to five slides. 


our slide show will be assessed on the: 
* ideas and information 
* organisation and design 
* use of software features, 


(Source: Australian Ministerial Council for Education, Early Childhood Development and Youth Affairs [MCEECDYA], 2008) 


Conventional Items 


These items will ask for students to select or construct their answers. The examples below are of 


the familiar constructed- and selected-response formats. 


e Constructed Response 


This task illustrates students’ skill in using simulations in a problem-solving activity. The stu- 
dent constructs a response by manipulating force arrows in a simulation-based scenario in which an 
emergency rescue truck must deal with various problems along a fire road in a forest. The student 
constructs text-based explanations of the balanced and unbalanced forces acting on the truck. While 
designed to be used as a middle school science item about force and motion, such a simulation could 
be adapted for the NAEP Technology and Engineering Literacy Assessment to study how the design 
of the technological system (transportation) affects the environment positively by making it possi- 
ble to contain forest fires and rescue people, and also negatively because of the cutting of trees and 
the disruption of wildlife habitat. This task also illustrates an interactive item in which students can 
manipulate different factors in a simulated environment and then provide responses to questions 
related to the scenario. For the NAEP Technology and Engineering Literacy Assessment, a series of 


items could call on the practices of Understanding Technological Principles (for example, describing 
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positive and negative effects of roads in the forest) and Developing Solutions and Achieving Goals 


(for example, designing a fire road that would not cross wildlife migration corridors). 


Context: Transportation 
Topic: Emergency Rescue 


Target Level: Grade 8 


In this challenge you were asked to draw a force that keeps the speed of the truck constant. 


How do balanced forces affect the speed of an object? 


How do unbalanced forces affect the speed of an object? 


(Source: Quellmalz, Timms, and Buckley, 2009) 


e Selected Response—Multiple Choice 


In this conventional multiple-choice item, the student selects an answer that describes the effect 
of fossil fuel on the natural environment, employing the practice of Understanding Technologi- 
cal Principles to analyze positive and negative effects. This item assesses student understanding of 
the interaction of a technology (energy production) and climate change. The combination of the 
two can be thought of as a system, with each affecting the other, and the item below also assesses a 
student's system thinking, albeit on a relatively simple level. Similar items in the NAEP Technology 


and Engineering Literacy Assessment could focus on the effects of technology on the environment. 
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Context: Natural World 
Topic: Climate Change 
Target Level: Grade 8 


The burning of fossil fuels has increased the carbon dioxide content of the 
atmosphere. What is the possible effect that the increased amount of carbon 
dioxide is likely to have on our planet? 


(A) A warmer climate 

A cooler climate 

©) Lower relative humidity 

(©) More ozone in the atmosphere 


(Source: Trends in International Mathematics and Science Study, 2003) 


Practices Applied in Design and Systems 


Assessment targets in the area of Design and Systems relate to the nature of technology, the 
engineering design process, maintenance and troubleshooting, and systems thinking. The sections 
below describe how the three crosscutting practices apply to targets in the area of Design and 


Systems. 


Understanding Technological Principles 


Technological principles for Design and Systems in chapter two specify for this assessment the 
core understandings that students should have about the different types of technologies, processes 
for designing technologies, approaches to preventing failures, and how components of technological 
systems interrelate. The practices for Understanding Technological Principles in these areas could, for 
example, ask students to draw on their knowledge to identify examples of technologies, components 
of the design process, components of a system, or maintenance and troubleshooting methods. Stu- 
dents could be asked to explain the relationship among technologies in a system, analyze the compo- 


nents of a system, recognize design constraints, or evaluate alternative representations of a system. 


Developing Solutions and Achieving Goals 


Problem-solving is a major part of the engineering design process. Thus there are many oppor- 
tunities for students to demonstrate their problem-solving skills in assessment tasks for this area. 
Such tasks could require them to develop designs, to propose or critique solutions to problems after 
being given criteria and constraints, to select appropriate resources by considering trade-offs, to 


construct and test a model or prototype, to troubleshoot systems and applications, or to determine 
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the consequences of making a change in a system. Students must use their understandings about 
the technological principles specified for Design and Systems in chapter two as they plan, try out, 


critique, and revise attempted solutions. 


Communicating and Collaborating 


Communication and collaboration practices are integral to achieving the goals of technolog- 

ical design and systems. Students can demonstrate teamwork in tasks where design assignments 

are distributed among team members, progress and results are integrated and shared, and products 
are presented jointly. Designs and the design process can be represented in visual and verbal forms. 
Students can create instructions for system assembly and prepare documentation of a procedure for 
maintaining a system. Students must use their understandings of the technological principles spec- 
ified for both Design and Systems and ICT in chapter two to use tools and strategies to communi- 
cate and collaborate. Because students are completing the assessment individually on the computer, 


collaborators will be virtual—that is, presented by the computer. 


Table 3.3 offers examples of how the three practices can be applied to assessment targets for 
Design and Systems to generate tasks and items for the middle school level. The key principles and 
grade 8 targets were selected from tables 2.6-2.9 in chapter two. The chapter two tables present 
simpler versions of the targets for grade 4 and more challenging versions of the targets for grade 12. 


Again, these are sample ideas for items and tasks and will not be used in the actual assessment. 
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Table 3.3 Examples of grade 8 tasks representing practices in each subarea 


of Design and Systems 
A. Nature of B. Engineering C. Systems D. Maintenance 
Technology Design Thinking and Trouble- 
shooting 
Selected Technologies Requirements for | All technological Tools and 
Principles developed for a design challenge | systems have parts | machines must 
one purpose are include the criteria | that work together | undergo regular 
sometimes adapted | for success and to accomplish a maintenance to 
to serve other the constraints or _| goal. ensure their proper 
purposes. limits that cannot functioning. 
Technological be violated in a 
Deacicee development solution. 
involves creative 
thinking. 
Describe the prop- | List three import- | How do the phys- | Why do Bill and 
erties of a spring ant criteria fora ical and human Sally oil their 
; that inspired the device that will components of the | bike chains and 
Understanding invention of the toast bread, and school cafeteria axles and check 
Technological Slinky. (animation) | justify each one. food service system | the brakes each 
Principles work together? month? What may 
(animation) happen if they do 
not? 
Given acollection | Design a process Design modi- Bill’s new bike 
of objects, design _| to serve 50 slices fications to the gears are not work- 
a new toy (for of warm toast in cafeteria for people | ing correctly. What 
Developing example, for a 5 minutes, given with a particular should he do? 
Solutions and baby, young child). | specific equipment | physical disabil- 
Achieving Goals What are the and resources. ity (for example, 
criteria for a toy, vision, mobility). 
and how does your (animation) 
design meet them? 
Select a team of How would an Collaborate with Bill, Sally, and 
people who could | industrial toaster a virtual group of _ | many other stu- 
design and build _| salesperson develop | people to redesign | dents would like 
a new toy fora talking points for | the tasks of the to ride their bikes 
nee 5-year-old, and selling a particular | cafeteria staff, given | to school. Present 
LOTS EES justify the choices. | toaster to a given the new machines. | a design for a bike 
and Collaborating Work individually, | restaurant? parking lot at the 
or collaborate with school. 
a virtual person 


to make your 
selections. 
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Illustrative Tasks and Items 


These items from existing tests were selected as examples that could be adapted for assessing 
targets in Design and Systems. Each item is analyzed for the ways in which it would call on the 


practices. 
Scenario-Based Item Set 


In this example, students are asked a series of questions related to a simulation of a nuclear reac- 
tor. In the NAEP Technology and Engineering Literacy Assessment, the questions might relate to 
all three practices. Students may be asked to demonstrate the practice of Understanding Technologi- 
cal Principles by identifying the inputs and outputs of the system or analyzing the potential hazards. 
Students might employ the practice of Developing Solutions and Achieving Goals in items asking 
for results of investigations based on manipulating the simulation to find safe levels of temperature 
and power. The practice of Communicating and Collaborating could be elicited in tasks involving 


virtual (that is, computer-generated) experts in preparing a report of findings. 


Context: Energy 
Topic: Nuclear Reactor 
Target Level: Grade 8 


Question 13: Nuclear Power Plant 
In the nuclear power plant simulation below, the number of nuclear interactions produced by the reactor is regulated oy the 


Control Rods. The power output is regulated by the Generator Valve. The Control Rods and the Generator Valve can be 
controlled oy cragging the sliders up and down. 


Contral Rods = Generator Velve 


Safety Valve 


Set the generator valve at 60%. HOW Tar (%) should the contro! rods be lied tor tne power plantto supply a continuous OUTOUT oF 
800 Megawatts (MVVi without the safety valve opening? 


© 85%. 
© 70%. 
© 85% 


© 100%. 


(Source: Organisation for Economic Co-operation and Development, 2005) 
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Conventional Items 


These items will ask for students to select or construct their answers. The examples below are of 


the familiar constructed- and selected-response formats. 


e Constructed Response 


Similar to the previous example, this item represents a sample task in which students must use 
their knowledge about the engineering design process to answer a constructed-response item. The 
practice of Understanding Technological Principles is measured by this computer-based item. In this 
item, students use various tools to explore the factors that affect plant growth in a greenhouse. In 
the NAEP Technology and Engineering Literacy Assessment, a similar item might ask students to 


evaluate different greenhouse designs in terms of their effects on plant growth. 
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Context: Plant Growth 
Topic: Designing a Greenhouse 
Target Level: Grade 12 


Plant qreomth 


Sarita and Tom want to 
check how much difference 


Page 1 Page 2 Page 3 Page 4) 


the light and warmth can 10°C 20°C 30°C 40°C 
make. 
Ve 
They decide to make one D is 
end of the greenhouse much a 
hotter than the other, and one 
side with much more light Quite 
than the other. Dark 
They put two plants in pots. 
One plant is to be grown as Quite 
fast as possible, the other Bright 
as slowly as possible. 
Very 
Bright 


Drag each pot to the correct part of the 
greenhouse. 


The bar chart shows how 
much the plants grew each 
day at different 
temperatures. 


Click on the slider to see 
how the amount of light 
affects the bar chart. 


Tom says that the plants 
will grow faster if they keep 
them as hot as possible. 


Is Tom right ? Click Yes or No: 


Q Explain your answer in you work book. 


(Source: Adapted from Ripley, 2009) 


83 | 


e Selected Response-Multiple Choice 


The following item is another conventional item type—in this case, multiple choice. Students 
would employ the practice of Understanding Technological Principles by demonstrating their 


knowledge about the engineering design process. 


Context: Product Design 
Topic: The Design Process 
Target Level: Grade 12 


An assignment for the design of an emergency light is described in the box below. 


Design and build an emergency light. It must have its own battery, provide a 
bright light, and have a built-in charger. It must be capable of operating while 
being charged. It must have a watertight enclosure with a handle on top. 


After receiving this assignment, which of the following most likely would be the next 
step in the engineering design process? 
A. testing and evaluation 
B. selecting the best solution 
J C. investigation and research 


D. construction of a prototype 


(Source: Massachusetts Department of Elementary and Secondary Education, 2009) 


Practices Applied in Information and Communication Technology (ICT) 


ICT literacy includes the capability to communicate ideas and solutions, to collaborate with 
peers and experts, to conduct research, to investigate solutions to academic and real-world problems, 
to find ways to meet the ever-changing needs of society, to properly acknowledge the source of ideas 
and information, and to select and use appropriate digital tools. The sections below describe how 


the three practices apply to the ICT assessment targets. 


Understanding Technological Principles 


The principles in the ICT assessment area involve understanding the variety of ICT tools and 
how and when they can be used to accomplish a wide variety of tasks in school and in practical, out- 


of-school situations. Students will need to know about the general features and functions of types 
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of ICT tools, which tools are appropriate for particular purposes, and to understand criteria for 


determining if the tools were used appropriately and well. 


Developing Solutions and Achieving Goals 


ICT tools can be employed to support problem-solving and achievement of goals in all three of 
the technology and engineering literacy areas. The types of problems addressed in the ICT assess- 
ment area relate to the selection and use of appropriate tools to achieve goals related to information 
research, investigating problems, meeting the needs of society, constructing and exchanging infor- 
mation and ideas, and acknowledging ideas and information. ICT problem-solving practices could 
be elicited by tasks and items asking students to select and use applications effectively and produc- 
tively; to access and use information and data to solve a problem or achieve a goal; to use ICT tools 
to solve a problem or achieve a goal; or to use ICT tools to plan an approach to solving a problem, 
represent data, analyze results, and summarize and present findings. Students must wse their under- 
standings about the features and functions of ICT tools in order to apply the tools appropriately and 


effectively to develop solutions and achieve goals for given tasks and problems. 


Communicating and Collaborating 


ICT capabilities rely heavily on students’ command of communication and collaboration skills. 
Students can be asked to demonstrate the capability to contribute effectively to a body of knowledge 
or to take part in group deliberations through social media (simulated in the assessment) and the use 
of other contemporary communication tools and structures. Students can be asked to investigate a 
problem or pursue a goal as individuals or with a group (virtually), to integrate input (provided in 
the task) from multiple collaborators (virtual) who are peers or experts, and to reach consensus. Stu- 
dents can integrate feedback from others, provide constructive criticism, and communicate to mul- 
tiple audiences using a variety of media and genres. Findings can be represented in a variety of ways, 
such as diagrams, tables, graphs, and digital media. To communicate and collaborate effectively, 
students will need to use their understandings about features and functions of communication and 


collaboration tools and apply their knowledge and understanding of the ICT tools and strategies. 


Table 3.4 provides some examples of how the three practices can be applied to assessment targets 
for Information and Communication Technologies to generate tasks and items at the middle school 
level. The key principles and grade 8 targets were selected from tables 2.10-2.14 in chapter two. 
Those tables further describe simpler targets at grade 4 or more complex targets at grade 12. These 


are sample ideas for items and tasks and will not be used in the actual assessment. 
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Table 3.4 Examples of grade 8 tasks representing practices that apply to each subarea 
of Information and Communication Technologies (ICT) 


A. Construction | B. Information C. Investigation | D. Acknowledg- | E. Selection and 

and Exchange Research of Academic ment of Ideas Use of Digital 

of Ideas and and Real-World | and Information Tools 
Solutions Problems 

Selected Digital tools offer | Important Digital tools can | There are multiple | Knowledge about 
Principles many options strategies for be very helpfulin | guiding principles | the common uses 
for formal and ensuring quality generating ideas (laws, policies, of readily available 

nonformal expres- | of information and solving prob- | and guidelines) digital tools sup- 


sion in nearly 


include 1) assess- 


lems in academic 


that govern the 


ports effective tool 


every academic ing the source of __| subjects as well use of ideas and selection. 
and professional information and as in research- information. 
discipline. 2) using multiple | ing practical 
sources to verify problems. 
the information 
in question. Infor- 
mation can be 
distorted, exagger- 
ated, or otherwise 
Practices F 
misrepresented. 
Describe how From among a Identify the Select the example Critique a digital 
graphics, text, and | group of Web combination of that shows an tool suggested 
Understanding tables convey a pages select the data tools that appropriate way for designing an 
Technological | message. particular one will best present to give credit to online story. 
Principles that contains a particular set of | another student's 
an example of data. graphic. 
exaggeration. 
Develop an Create a digital Use simulations Identify which Predict trends in 
: online survey for | story aboutahis- | and visualizations | online images rates of software 
Devcon ns elementary school | torical period by to describe the can legally be piracy based on 
stg and students concern- | choosing images rate ofdeforesta- | usedinastudent | provided data. 
Achieving Goals ing the design of a | of art from the tion in Brazil. presentation. 
new playground. | period. 
Respond to Aska virtual Enter costs from Post a copy- Use two digital 
suggestions from | collaborator for several sources right-free image tools to create 
two virtual (com- | help on devel- and communicate | toa website and a public service 
puter-generated) | opinga digital to the principal communicate to announcement on 
collaborators presentation. the most eco- friends that it is software piracy. 
Communicating explaining why nomical printer available. 
and Collaborating the search results for school play 
of only one of posters. 


the collaborators 
have sufficient 
information for 
the report. 


| 86 


Illustrative Tasks and Items 


The following tasks and items illustrate how the three practices apply to ICT assessment targets. 


The practices may be elicited by extended, scenario-based task and item sets or by separate items. 
Scenario-Based Item Sets 


In this simulation, students navigate among a file manager, an email client, a Web browser, a 
word processor, and a spreadsheet to make a travel brochure for a fictional town, Pepford. They are 
assessed on how they use these ICT tools to accomplish the task, not on the quality of the brochure. 
The process is more important than the outcome. Students use the practice of Understanding Tech- 
nological Principles by calling on their knowledge of which ICT tools to use to accomplish the goal 
and how to use them. Students also apply the practice of Developing Solutions and Achieving Goals 
as they conduct their information research. The practice of Communicating and Collaborating is 
not assessed directly in this task, although it could be added by including tasks assessing the quality 
of the brochure for communicating to the intended audience and tasks involving the assignment of 
information gathering from virtual (computer-generated) collaborators and the integration of their 
(virtual) input into the brochure. 


Context: Travel and Tourism 
Topic: Promotional Brochure 


Target Level: Grade 8 


Connected | Rakesh Sahu :- Tourist Information Centre && 
o ~ 
‘T  Pepford Tourist Information Centre - Home page 


File Egit View Favourites Tools 


SD MIMAMACIETO 


Welcome 


Home 


Welcome to the official website of Pepford Tourist Information Centre 

This is the best ane stop service for visitor and tourist information in the 

town, handling thousands of enquiries every year, There is a warm 

welcome waiting for you in Pepford. No one knows more about the town 

and what is going on than we do. Before you make your plans or think 

Contact us about where to stay, come and pop into the central visitor building to 
2ontac find out more. You'll be glad you did. 

Events Pepford is situated in the county of Waltonshire. The beautiful river Pep Spreadsheet 

runs through the south west side of the town. We are easily accessible 

Where to Visit by road and rail, so getting here for a short break couldn't be easier. 

Pepford is 2 lively and thnving community, which offers an interesting mix 

of theatre, cinema, gallery, festivals, restaurants, sporting facilities and 

good quality shops. 


Staff Page 


& holiday in Pepford will leave you feeling refreshed, relaxed and 
recharged - even if you're on the go 24 hours 3 day! All year round there 
is something for everyone. 


Visit in the summer and experience the fun of the festivals. 


(Source: Ripley, 2009) 
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Conventional Items 


These items will ask for students to select or construct their answers. The examples below are of 


the familiar constructed- and selected-response formats. 


e Constructed and Selected Response 


In this task, students are presented with a list of websites from the results of a Web search. Stu- 
dents must decide which website has the most reliable information and justify their response. The 
practice of Understanding Technological Principles is applied to students’ knowledge of criteria for 
how to judge the credibility and quality of information sources—one of the ICT targets in Informa- 


tion Research. 


Context: Medicine 
Topic: Web Search 
Target Level: Grade 8 


Section 2 


DIRECTIONS: 

ched the Internet for general 

out treating bee stings and found the 
sites shown 


Pick one site that you think has reliable information 
about treating b S. Type the letter of that site 


(A-J) in the box below 


xplain why you chose this site 
ype your answer below 


(Source: Lennon et al., 2003) 


‘PISA-ICT ‘Literacy Assessment - 


in Srl 4 Advance Tools Preferences — Lanquage Tools 
MeO wi@ (e& 
. treat bee stings Search 


Bee Venom Therapy (Bee Stings) | AHealthyMe.com 
Find out whether sul g yourself to bee stings can help ease the pain 
bee venom to succ ly treat rheumatoid arthritis 

sv ahealthyme. com/topic/beevenom - 49k - Cached - Similar pages 


Families for Natural Living 
Have treatment brought to them When | was a child, my mother always 
bee stings and other venomous 


familiesfornaturalliving org/herbs/summ - Cached - Similar pages 


for Bee Stings __ 
a mplex is Great for Poison Ivy, Poison Oak, Hives, 
g or treating a health proble 
a= 41k - Cached - Simila 


MEDLINEplus Medical Encyclopedia: Insect bites or stings 

develop severe allergic reactions (anaphylaxis) to insect stings. ... the sting 
is from a honey bee, rernove the ge treat the victim 
nw. nim nih. gowmedlineplus/ency/article/OO0L tr - 43k - Cached - Similar 


pages 


‘stings? What insect 
California. Bee 
hed - Similar pages 
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e Selected Response—Multiple Choice 


This multiple-choice item also assesses a student’s capability to evaluate websites, an example of 
the practice, Understanding Technological Principles. This example also demonstrates a comput- 


er-based selected-response format. 


Context: The Internet 
Topic: Website Search 
Target Level: Grade 12 


BD Search Web - search engine - Search Web - Microsoft Internet Explorer 


fe ES_Yew_favortos_Toes_teb 


ie) hetp:|/searchweb.comy_1_O4PUTDO31 WIMZFD__websrch. barweb.3/dog/resuks?otmpl=dog/webresuks himaghw=search%20%2_* kg VG 


Web mages News More 
Search Qweb SSS sas ss 
Atvanced Search 


gran YICTO A ZEXPOQ Fik Format: PDF/Adobe Actobat - v 

«. Through our Game Pian strategies. the Sracks Government is strongly supporting what is widely recognized as the nation's 
computer game capital, Australia’s leading developer of video games for personal... 
www minv vic gov.awweb Simav nefImageLookupMediaQ2 2004/$file/10May04% WE3% WExpo pdf'- Sivitar pages 


and behaviour - Stepfamily Zone For cussion Area 
= 33 is also showing more interest in the video games and |... While playing one of hese games, he punched our friends... He 
has never been aggressive or violent soi... 
nip JAverw stepfamby asn au/cgibay% 20BatyBlah pleb=gen2004 display, m=10941 86800% 20-% 205k 
- Similar pages 


[PDF] Video Games S Youth Violence 

File Format POFSAdobe Acrobat - View as HTML 

Page 1. Video Games & Youth Violence. EXAMINING THE FACTS INTERACTIVE DIGITAL SOFTWARE ASSOCIATION 
no evidence to support claims that there is a link between violence mM ideo games and in 

vweww Iheesa comiDSAfinal pdf - Simfbar pages 


fectron Your Chi 
 @vidence thatviolent video games can cause an increase in violent behaviour and other social problerns in children. Taken 
Together, the research suggests that... 
vw wicparenting. gov au/staging! files/Electronicgames2002¥1 htm! - Similar pages 


) | 
Rsrat| [= 


Click on the link that is likely to contain information from the most RELIABLE and 
RELEVANT source, 


| (@ Internet 


(Source: Australian MCEECDYA, 2008) 
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Contexts 


Technology and engineering literacy requires not just that students know about technology but 
also that they are able to recognize the technologies around them, understand the complex relation- 
ship between technology and its effects on society, and use technological principles and tools to 
develop solutions to problems and meet goals. Consequently, NAEP Technology and Engineering 
Literacy Assessment items will measure students’ technology and engineering literacy in the context 
of relevant societal issues, design problems, and school and community goals. As the three areas of 
technological literacy to be measured by NAEP tend to focus on somewhat different types of issues, 
problems, and goals, the contexts and situations that will frame the technology assessment items in 


these areas will differ somewhat as well. 


Contexts in Technology and Society 


The complex and multifaceted interactions between technology and society often manifest 
themselves in unexpected and unpredictable ways as new technologies are used in particular con- 
texts or situations. A new technology may succeed in meeting the need that it was intended to meet 
and bring about far-reaching benefits, but it may also have negative, unintended consequences. For 
example, mobile communication devices have transformed business and personal interactions, yet 
a large number of traffic accidents have been blamed on drivers using these devices while operating 
their vehicles. Similarly, farming practices have increased crop production, but they have also risked 
contaminating sources of groundwater. Such issues can clearly be used as contexts for NAEP Tech- 
nology and Engineering Literacy Assessment items (as illustrated in ISTE’s NETSYS, the frame- 
work of the Partnership for 21st Century Skills, and ITEEA’s Standards for Technology Literacy). 
The contexts for tasks and items in the area of Technology and Society may include such technolo- 
gies with positive and negative effects and may also present ways that technology improves people’s 
lives, such as water purification, sewage treatment, and medicine, or the various ways that people 
regularly interact with technology, from brushing teeth in the morning to crawling into a warm, 
comfortable bed at night. The following are examples of topics in the contexts of health, energy, and 
electronic communications that could be used to generate assessment tasks and items for sample 
targets in the three subareas of Technology and Society for grade 8: 


e Agriculture and health contexts: water as a scarce resource; 
e Energy context: wind turbines for homes; and 


e Electronic communications context: personal communication devices. 
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Table 3.5 Examples of how different contexts may be used to generate 
tasks and items for Technology and Society for grade 8 


A. Interaction of | B. Effects of Tech- C. Effects of D. Ethics, 
Technology and nology on the Technology on Equity, and 
Humans Natural World the World of Responsibility 
Information and 
Knowledge 
Society drives tech- | Some technolog- | Information tech- | Technology by 


nological change, | ical decisions put _| nology is evolving _| itself is neither 
while changing environmental and | rapidly, enabling good nor bad, but 
technologies in economic concerns | ever-increasing its use may affect 
turn shape society. | in competition amounts of infor- | others. 
with one another, | mation and data to 
Key Principles while others have _ | be stored, man- 
positive effects aged, enhanced, 
for both the analyzed, and 
economy andthe | accessed through 
environment. a wide array of 
devices in various 
media formats. 
What societal What issues need _| Find two reports How might the 
Context: needs drove the to be addressed describing alterna- | decision to divert 
Agriculture and | changes made toa | toensure that the | tive water purifica- | water from the 
Health: Water asa | river’s natural flow? | water system stays | tion methods. rivers affect farmers 
Resource healthy? and small towns 


downstream? 


Context: Energy: 


Describe the 
positive and neg- 
ative impacts that 


Compare and 
contrast the envi- 
ronmental and eco- 


Compare the 
persuasiveness of 
two multimedia 


Describe a process 
for citizens to 
evaluate the effect 


; ‘ residential wind nomic impacts of | presentations on that wind tur- 
Wind Turbine ; ; ; ; ; ; ; ; ; 
turbines might wind turbines with | alternative wind bines might have 
have on society. other potential turbine designs. on others in the 
sources of energy. community. 
What are the Describe the effect | Describe ways that | What might 
positive and neg- of video evidence personal commu- the effect be of 
Cc ative effects that of environmen- nication devices allowing personal 
ontext: ; ‘ oe 
: personal commu- _| tal destruction provide access to communication 
Electronic ie ; se ; ; ; 
ae nication devices on society’s information and devices to be used 
Communication 


may have on 
traditional human 


communication? 


awareness of the 
global impact of 
pollution. 


expertise. 


in school? 
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Contexts in Design and Systems 


Nearly all of the products and processes in human society result from the development of one or 
more kinds of technology. Homes, factories, and farmhouses are built using construction technolo- 
gies. Fruits and vegetables are grown and processed using agricultural technologies and are brought 
to market and to the dinner table with transportation technologies. Methods of extracting and using 
fuels to produce power involve energy and power technologies, and the tools and processes used by 
doctors, nurses, and pharmacists are a part of medical technologies. Although these technologies can 
be classified in various ways, in order to provide guidance to item writers, this framework identifies 
the following technology areas that can be used as contexts to measure students’ understanding of 
design and systems (drawn primarily from ITEEA, 2007): 


e Agricultural and related biotechnologies; 

e Construction technologies; 

e Energy and power technologies; 

e Information and communication technologies; 
e Materials and manufacturing; 

e Medical technologies; and 


e ‘Transportation technologies. 


The section on the next page presents potential scenario topics placed in contexts from the types 
of technologies listed above. The table illustrates how the topics in these contexts can be used to 
generate tasks and items in the four subareas of Design and Systems for grade 8. The scenarios would 


be simpler for grade 4 and more complex for grade 12. 
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Table 3.6 Examples of how different contexts may be used to generate 


tasks and items for Design and Systems for grade 8 


A. Nature of 
Technology 


Scientists are con- 
cerned with what 
exists in nature; 

engineers modify 


B. Engineering 
Design 


Requirements for 
a design challenge 
include the criteria 
for success, or goals 


C. Systems 
Thinking 


Systems may 
include subsystems 
and may interact 
with other systems. 


D. Maintenance 
and 
Troubleshooting 


Tools and machines 
must undergo regu- 
lar maintenance to 

ensure their proper 


Key Principles | natural materials to | to be achieved, and | Systems may also be | functioning. 
meet human needs | theconstraintsor | embedded within 
and wants. limits that cannot __| larger systems. 
be violated in a 
solution. 
How have trans- Propose two ways What are the What problems 
portation methods | to modify an inter- | advantages of con- | might occur if 
Context: changed over time? | section to make it tainer cargo ships engines are not 
Transportation safer. over other ways to__| oiled periodically? 
transport goods to 
market? 
What were the Identify the require- | Name several What procedures 
technological ments for a pros- elements of the would you recom- 


Context: Medi- 


advances that 


thetic arm that will 


nation’s medical 


mend for main- 


cal Technology | allowed medical enable a person to _| system and describe | taining the school’s 
researchers to play tennis. how they are first-aid kits? 
develop vaccines? related. 
What natural Compare the Using the simu- Using the simu- 
constraints exist in | aesthetic qualities lation model of a lation model of a 
a city that might of the two types of | residential wind residential wind 
Context: ; : eo . . : 
: cause ahomeowner | wind turbines (ver- | turbine, identify turbine, describe 
Energy: Wind ; ; 
Turbi to choose wind tical or horizontal). | the goals, inputs, which parts of the 
urbine 
power over other processes, outputs, | machine would 
“green” energy and feedback and require the most 
alternatives? control features. maintenance. 
‘Trace the evolution | Compare the trade- | Explain two ways Describe a set of 
Cc ; of features on early | offs of functions in which personal troubleshooting 
ontext: 


Information and 
Communication 


Technologies 


cellphones com- 
pared with current 
smartphones. 


available in two 
specific devices 
designed for a 
workplace or per- 
sonal use. 


communication 
devices can work 
together for a team 
to achieve its proj- 
ect goal. 


steps that would 

be appropriate for 
analyzing a problem 
with a printer. 
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It is important to note that students are not expected to be familiar with the specific compo- 
nents and working details of any particular technology. For example, they will not be tested on their 
knowledge of genetic engineering, an important biotechnology, or on their understanding of energy 
and power or networking technologies. While these topics may be used to provide the context for 
test items, the information required for students to respond to the test questions will be provided 
in the scenario or background of the question. Students will be tested on the broad set of princi- 
ples concerning design and systems and capabilities described in chapter two. However, one of the 
technologies from the list in the previous section has been chosen for more emphasis in the 2018 
NAEP Technology and Engineering Literacy Framework, and that is Information and Communica- 
tion Technology. 


Contexts in Information and Communication Technology (ICT) 


In contrast to other types of technologies, students will be expected to be fluent in the use of 
information and communication technologies, as described in the tables in chapter two. The reason 
for this additional attention to ICT is that it is pervasive in society, and some level of technology 
and engineering literacy is required for virtually every profession, in every school subject, and in all 
walks of life. Furthermore, it is likely that literacy with information and communication technolo- 


gies will become even more important in the decades ahead. 


Because of the ubiquity of ICT, it is difficult to describe all of the particular contexts for items 
that NAEP will design to assess students’ knowledge of it and their capabilities to put it to use. ICT 
knowledge and skills can be applied in the context of developing and using any of the technologies 
included in table 3.7, and it can be applied to any of the ways that technology interacts with society. 
ICT principles and tools should be a part of every person’s set of capabilities in and out of school for 
solving problems or working to meet a goal. ICT tools have become integral supports for learning 
school subjects. People who are literate in technology and engineering should be able to select and 
use technological tools to research a period in history, compare cultures, collect and display data 
in a scientific investigation, develop a story or presentation, or produce a work of art. The types of 
scenarios used to assess students’ knowledge and skills in this area will require that the item provide 
an opportunity for students to demonstrate their understanding of and capabilities to use ICT to 
address goals and problems in Technology and Society, in Design and Systems, and, more generally, 
in various disciplines, and in real-world, practical applications. The following table illustrates how 
topics set in different contexts can be used to generate tasks and items for targets in the five subareas 


of ICT. 
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Table 3.7 Examples of how different contexts may be used to generate tasks and items 
for Information and Communication Technology (ICT) for grade 8 


A. Construction and 
Exchange of Ideas 
and Solutions 


B. Information 
Research 


C. Investigation 
of Problems 


D. Acknowledgment 
of Ideas and 


Information 


E. Selection and 
Use of Digital Tools 


Digital tools offer 
many options for 
formal and nonfor- 
mal expression in 


Important strategies 
for ensuring quality 
of information 

include 1) assessing 


Digital tools can 
be very helpful in 
generating ideas and 
solving problems in 


Fair use guidelines are 
designed to support 
the use of copyrighted 


materials for aca- 


A fundamental 
aspect of technolog- 
ical literacy is the 
possession of founda- 


ee nearly every academic | the source of infor- academic subjects as | demic purposes and tional ICT skills in 
and professional mation and 2) using | well as in researching | for journalism and the use of common 
discipline. multiple sources to practical problems. other forms of writing | productivity tools. 
verify the informa- and commentary. 
tion in question. 
Web pages have been | Synthesize data from | Watch an example Review examples Using articles and 
created to persuade a variety of sources video of a success- of media that simulated websites, 
people to move to (census, local econ- ful local campaign could be used for a create a media- 
Cente your community. Use | omy, demographics, | to declare a local presentation at alocal | rich presentation 
Piacdical a rubric to evaluate industry, history) to building asa historic | community meeting | designed to persuade 
Applicidons: mock-up samples of show the availability landmark. Answer and then make people to move to 
Tac Web pages, including of jobs in your town. | a series of questions decisions as to the your city. 
Community the appropriate use about the effective- appropriateness and 
feren of media to provide ness of the video. legality of using them. 
information to appro- 
priate audiences and 
also the appropriate 
citation of sources. 
Using information Using the provided | Using the simula- Examine a set of Using the informa- 
from the text and the | Internet searchand | tion modelandthe | slides and associated | tion provided in the 
provided slide-mak- slide-making tools, available city data, resources from a wind | text, choose from the 
ing tools, design a research and create a | determine which turbine manufac- available digital tools 
presentation that presentation compar- | citiesin the United | tureranddetermine | and create a multi- 
Context: presents the positive | ing and contrasting | States would be how best to give media information 
Energy: and negative impacts | the benefits of resi- most appropriate for | appropriate credit packet to promote 
Wind of residential wind dential wind turbines | installing residen- for the information residential wind 
Turbine turbines to a group of | with other “green” tial wind turbines. and images used ina __| turbines. 
homeowners. energy alternatives. | Be sure to use data presentation to pro- 
from the simulation | mote residential wind 
to support your turbines. 
conclusions. 
Taking the role ofa Conduct research in | Working with online | Develop credits and | Explain how the 
student during the online libraries (for | archives, compare acknowledgments for | ICT tools you used 
Civil War, refer to example, Library of | the hardship endured | your online sources. contributed to a 
Context: examples from the Congress) synthe- by civilians during presentation on the 
School online archives to sizing news accounts | the First Battle of Battle of Gettysburg. 
Subjects compose aletterand |andpicturesonthe | Bull Run and the 
History photo essay describing | events and their Battle of Gettysburg. 
the First Battle of Bull | impacts. 
Run. 
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The intent of placing technology and engineering literacy assessment tasks and items in a range of 
contexts is to ensure that students have the opportunity to demonstrate what they know and can do 
across many types of problems and situations. By sampling student knowledge and skills in a variety 
of contexts and practices, the assessment will provide a broad and deep picture of technology and 


engineering literacy proficiencies. 
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CHAPTER FOUR 


OVERVIEW OF THE ASSESSMENT DESIGN 


Introduction 


This chapter provides an overview of the major components of the assessment design. It begins 
with a brief description of the 2018 NAEP Technology and Engineering Literacy Assessment and 
follows that with a discussion of the types of assessment tasks and items, how they can be used to 
measure student understanding and skills, how students will respond to these tasks, and how their 
responses will be monitored and evaluated. In addition, this chapter describes how the assessment 
should be balanced across the major assessment areas in technology and engineering literacy as well 
as across the practices. The types of items to be included in the assessment are described, and exam- 
ples are provided. English language learners and students with disabilities were recognized from the 
start as an important part of the student population to be included in the 2018 NAEP Technology 
and Engineering Literacy Assessment, and thus were considered as the assessment framework was 
developed. Concepts of universal design for assessment were considered as one way for the assess- 


ment to be more inclusive of all students. 


Overview of the NAEP Technology and Engineering Literacy Assessment 


In 2018, the NAEP Technology and Engineering Literacy Assessment will be conducted at the 
national level for grade 8. The assessment will include tasks and items sampled from the domain of 
technology and engineering literacy achievement identified by the intersection of the three major 
areas of technology and engineering literacy and the crosscutting practices. The assessment will be 
administered by computer and will be composed of sets of long scenarios, short/medium scenarios, 
and discrete items. Within each of these types of tasks there will be a variety of selected-response 
items and short and extended constructed-response items. Student responses will be measured both 
directly and, in the scenario-based tasks, through their interactions with simulated tools and their 


manipulation of components of systems. 


Types of Tasks and Items 
Allowing students to demonstrate the wide range of knowledge and skills detailed in the NAEP 


Technology and Engineering Literacy Assessment targets will require a departure from the typical 


assessment designs used in other NAEP content areas. Thus students will be asked to perform a 
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variety of actions using a diverse set of tools in the process of solving problems and meeting goals 
within rich, complex scenarios that reflect realistic situations. Consequently, this assessment will rely 
primarily on scenario-based assessment sets that test students through their interaction with multi- 
media tasks that include conventional item types, such as selected-response items, and also monitor 


student actions as they manipulate components of the systems and models that are presented as part 


of the task. 


The following sections describe in detail the scenario-based assessment sets and the sets of dis- 
crete, conventional items that will be developed for the NAEP Technology and Engineering Liter- 
acy Assessment. Note that the examples of items shown were drawn in spring 2010, reflecting what 
was available at the time of developing this framework for the first administration of the NAEP 
Technology and Engineering Literacy Assessment, conducted in 2014. It is expected that, as the use 
of such innovative items expands and technology advances, these examples will become dated, but 


they still illustrate the principle components that are important in this assessment. 


The assessment will be administered to a nationally representative sample of students to report 
on student achievement at the group level. The assessment is not designed to measure the perfor- 
mance of any individual student or school. To obtain reliable estimates across the population that is 
tested, a large pool of assessment items will be developed. That pool of items will be too large to give 
to any individual student, so subsets of items will be selected to administer to each individual stu- 
dent. The NAEP Technology and Engineering Literacy Assessment will be given in approximately 
60 minutes, with additional time for contextual questionnaire completion. The assessment sets that 
will be developed for the NAEP Technology and Engineering Literacy Assessment are described 


below. 


Scenario-Based Assessment Sets 


There will be scenario-based assessment sets of different lengths: short, medium, and long. The 
long scenarios will take students approximately 30 minutes. Medium scenarios will take students 
about 20 minutes to respond, and the short scenarios will take students about 10 minutes. The two 
types of scenarios have common characteristics, but they differ in complexity and in the number of 
embedded assessment tasks and items to which a student is asked to respond. Long scenarios will 
provide about 10 to 15 measures of performance, and the short scenarios will capture approximately 
5 to 10 measures of performance. Measures will include innovative measures of a student's interac- 
tion with aspects of the scenario as well as conventional selected-response items and short construct- 


ed-response items. The different measures are discussed later in this chapter. 
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Discrete Item Sets 


One of the challenges for this assessment is that the use of scenario-based assessment sets 
reduces the number of independent measures in the assessment as a whole. Because of their capabil- 
ity to replicate authentic situations examinees may encounter in their lives, scenarios have the poten- 
tial to provide a level of authenticity other types of assessment tasks cannot provide, which, in turn, 
may contribute to the validity of the entire assessment. At the same time, however, the choice to use 
these complex tasks reduces the number of measures that can be included in any one test and causes 
many of the measures to be interdependent because they are related to the same scenario. To coun- 
teract this interdependency and ensure reliability, the NAEP assessment of technology and engi- 
neering literacy will also include sets of discrete items that produce independent measures. Discrete 


item sets will include conventional selected-response items and short constructed-response items. 


Definitions of the Scenario—Based and Discrete Item Assessment Sets 


Scenario-Based Assessment Sets 


As this framework is written, the use of computer-based scenarios for assessment purposes is 
an emerging but growing area. The 2009 NAEP Science Framework called for the use of interactive 
computer tasks as part of its assessment, and in 2009, three long and six short interactive tasks were 
administered to national samples of students in grades 4, 8 and 12. Another set of interactive assess- 
ment tasks has been used in state tests of science achievement in an Enhanced Assessment Grants 
project funded by the U.S. Department of Education to determine how simulation-based scenarios 
in science might form part of district and state accountability systems. As assessment developers 
gain more experience in this emerging field, they will develop a better sense of how to create the 


tasks efficiently and how to ensure that the tests produce valid and reliable measures. 


Here a descriptive outline is provided of the main features of the scenarios that will be developed 
for the NAEP Technology and Engineering Literacy Assessment. At the beginning of the scenario, it 
is important to set the context for the activities in which the student will be involved. This introduc- 
tion provides a setting for the assessment tasks that, as far as possible, should reflect tasks that might 
be performed in society, either within an academic setting or outside of school. In addition, near the 
start of the scenario, a motivating question or goal will be introduced. This goal provides the driving 
rationale for the tasks that the student will perform, and it offers a storyline that helps define the 
relevance and coherence of the tasks and motivates the student to undertake them. The motivating 


goal might be to solve a particular problem or to achieve a certain goal within the scenario. 
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An advantage of computer delivery of the assessment is that the introductions to the scenarios 
can use appropriate multimedia to present the settings for the assessment tasks. As a result, there 
is less need for text and therefore less of a reading demand. The multimedia can include video 
segments or animations that the student observes, and it will also generally use text, numbers, and 
graphics to convey information necessary for the tasks to be accomplished. All of the representa- 
tions, such as graphics, video, or simulations, must be carefully chosen to serve a purpose in the 


assessment tasks, and none should be present simply for visual interest. 


One type of assessment scenario will include a representation of a system. Depending on the 
context for the particular scenario, this might be an engineering system such as an irrigation sys- 
tem or a dam. Whatever the system, it will have components that are dynamically related, so that a 
student can observe the role of a particular component (for example, watch what happens when a 
valve is opened in an irrigation system) or interact with a component (for example, by setting a value 
for a parameter or moving a part of the system) and see a resulting change of state in the system (for 


example, a rise in water levels or flooding of fields). 


A second type of scenario will lay out an overarching goal or problem that students will reach or 
solve by conducting various interrelated tasks. Such a goal might be, for instance, to develop a play 
about a historical period. Component tasks could involve searching for information about a famous 
incident in the period, analyzing the necessary ticket sales, and creating a playbill advertising the 


production. 


Within a scenario, students may be asked to select tools from a toolkit and use them within the 
system. Students might be asked, for example, to select a graphing or spreadsheet tool or to use a 
simulation. Various tools may be made available, depending on the scenario. Word-processing, tex- 
ting, or presentation tools might be available for communication tasks, for example, and Web design 
or page layout tools might be used for the presentation of large amounts of information. For some 
scenarios it might be appropriate to provide more specialized tools, such as computer-aided design, 


geographical information systems, or video editing tools. 


By interacting with the components of the system or task that are key elements for achieving 
the goal, students are able to respond to tasks that ask them to explore alternative outcomes, con- 
trol certain variables, and observe the resulting changes in the system. The students can observe and 
describe the patterns or characteristics of the outcomes and can interpret the feedback from the 
system. They can then evaluate the outcomes of the choices they made in manipulating the compo- 


nents of the system or in using particular tools, and, finally, they can form conclusions. 
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In some cases it might be necessary to simulate virtual features of real-world equipment that 
can be used within a scenario. For example, a temperature gauge might give feedback from a heating 
system, an anemometer might be used to measure wind speed in a scenario about wind turbines, or 
a table might be used in an ICT scenario requiring the collection of data about the types of sym- 
bolism in Shakespeare’s plays. Alternatively, graphics or images might be constructed or selected to 


communicate a design or idea. 


In providing tools in a scenario, it is necessary to determine which elements of a tool are nec- 
essary for the activities in the scenario and which features of the tool will be used by students. It 
is not necessary to provide or simulate a fully featured version of a tool. For example, only certain 
functions of a spreadsheet tool might be provided in order that students could take a table of data 
resulting from actions in the system and transform it into a graphical representation of their choice 
(a line graph, say, or a bar graph or pie chart). It would not be necessary to provide all of the other 
features of the spreadsheet tool, and, in fact, it would be distracting to students and produce mea- 


surement “noise.” 


Throughout their interaction with a system, students may be asked to use tools to find relevant 
resources; to communicate to others about their actions, decisions, or results (for example, texting a 
virtual team member); or, at the end, to convey their conclusions (for example, creating a slideshow 


presentation). 


Discrete Item Sets 


The discrete item sets will comprise approximately 10 to 15 stand-alone items in either selected- 
or constructed-response format to be completed within the 60 total minutes that each student will 
spend on the assessment tasks. These items would not be part of a complex scenario or related to one 
another. Each discrete item would provide a stimulus that presents enough information to answer 
the particular question posed in the stem of the item. Items in discrete sets will be selected-response 
items (for example, multiple choice) or short constructed-response items in which a student writes a 


text-based response. 


Descriptions of the Response Types Used in the Assessment Sets 


In conventional items on previous NAEP assessments, students have responded either by 
selecting the correct response from among a number of choices or else by writing a short or long 
text-based response to the questions posed. In the computer-based NAEP Technology and Engi- 
neering Literacy Assessment, with its scenario-based assessment sets, there are opportunities to 


greatly extend the ways in which a student can respond to an assessment task. Thus this assessment 
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can move beyond the old ways of thinking of response types in terms of simply multiple-choice or 


written responses and begin to consider new types of responses. 


In this assessment, three response types are used: short constructed response, long constructed 
response, and selected response. Although these are the same names as used in other NAEP assess- 
ment frameworks, in the context of the NAEP Technology and Engineering Literacy Assessment 


they have different and expanded meanings. These meanings are described in the following sections. 


Constructed Response 


Constructed responses are ones in which the student “constructs” the response rather than 
choosing a response from a limited choice of alternatives, as is the case with selected-response items. 
Constructed responses in the NAEP Technology and Engineering Literacy Assessment will include 
short constructed-response tasks and items as well as extended constructed-response tasks and 


items. These are described in detail in the following sections. 


Short Constructed Response 


Short constructed responses might be used in either the discrete-item assessment sets or in 
the scenario-based assessment sets. They generally require students to do such things as supply the 
correct word, phrase, or quantitative relationship in response to the question given in the item, to 
identify components or draw an arrow showing causal relationships, to illustrate with a brief exam- 
ple, or to write a concise explanation fora given situation or result. Thus students must generate the 
relevant information rather than simply recognize the correct answer from a set of given choices, as 
is the case in selected-response items. When used as part of a discrete item set, all of the background 


information needed to respond is contained within the stimulus material. 


The following is an example of a short constructed-response item that might be used in a dis- 


crete item set. In this computer-based item, students use a spreadsheet program to create a pie chart. 
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ue rf Format 
Dd S/B-\O-%-\E-\fiom - Bfpleseenws o-o-A- 
a fa fa ed @ BS | By QB) Pe Rept wth changes... End Review | 


Create and label a pie chart to display only the data for 2003, 
When you have finished, click on the ‘I've finished’ button. 


(Source: Australian MCEECDYA, 2008) 


Extended Constructed Response 


Extended constructed responses will be used in the long scenario-based assessment sets. In a 
scenario-based assessment set, the real-world scenario is developed and elaborated on as the student 
moves through the assessment set. As previously described, the introduction of the scenario will pro- 
vide context and motivation for the tasks in the assessment set. As the scenario builds, the student 
undertakes a series of tasks that combine to create the response. For example, a student might be 
asked to enter a search term to gather information about a famous composer and to request infor- 
mation from virtual team members. Students could vary the size of populations to test a model of 
a city’s transportation system, or, in a different scenario, they might be asked to construct a wind 
turbine from a set of virtual components in which there are several combinations of turbine blades 


and generators. 
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Additional measures of the students’ responses can be made by capturing data about which 
combinations of components the students selected, whether they covered all possible combinations, 
and what data they chose to record from their tests of the components. A follow-on task might ask 
the students to select different types of graphic representations for the tabulated data they captured. 
Observing whether they have selected an appropriate type of graph provides additional information 
about how they use data analysis tools. 


Finally, the students could be asked to interpret their data, make a recommendation for the 
best combination of turbine blade and generator, and justify their choice in a short written (typed) 


response. In this way, both the task and the response are extended. 


Thus, unlike short constructed-response items in which all the information to answer a partic- 
ular task is contained in a single stimulus, the information necessary to answer an extended con- 
structed response is contained in several parts of the overall task. In this example, it would not be 
possible for a student to make recommendations about which combination of blades and generators 


is best without having done all parts of the previous tasks. 


Designing extended constructed-response tasks presents certain challenges. Enough informa- 
tion must be provided in the scenario to allow the student to perform well-defined, meaningful 
tasks that yield measurable evidence about whether the student possesses the knowledge and skills 
defined in the assessment targets. Another challenge is to ensure that the dependencies among the 
tasks that a student performs within an extended response are minimized. For example, in the wind 
turbine scenario described above, students could run tests of combinations of certain turbine blades 
and generators, and their responses could be assessed. Then, the students could be given data from 
another set of tests of different blades and generators that someone else did and asked to interpret 
those data. In this way the dependency between a student’s own data-gathering and the data analysis 
stage is minimized. The goal is to make sure that mistakes or deficiencies in the first part of the task 
are not carried forward into the second task, thereby giving all students the same opportunity to 


show their data analysis skills, regardless of how well they did on the first task. 


In the following simulation, students are given the scenario of a population of small birds— 
chortlers—whose population is declining. The students are asked to use various tools to analyze data 
in a variety of ways and to determine some possible causes for the population decrease so that they 
can then present their findings on the impacts on the chortlers. The multiple developments of the 
graphs are extended-constructed responses, as is the development of the conclusions to be presented, 


which in this example are not written on the computer. 


| 104 


Chartlere 


Paget 


The chortler is a small woodland bird which has been declining in numbers 
recently. Here are some ideas why chortlers are becoming less common: 


@ More air pollution 

@ High summer rainfall 

@ Increasing numbers of hawks 
120 woodland sites are surveyed. 


The number of chortlers and hawks are 
recorded as a percentage of the numbers 
found 5 years ago. 


The average summer rainfall and air pollution 
levels have been recorded for the last 5 years. 


Go to page 2 and explore the survey data. 


Chartlers 


Click on the buttons below to present the data in different ways. 
Browse & Sort Data Draw Graphs Group Data 


Drag these labels to the 
graph axes to choose 
what to plot. 


=Hawk Numbers (%) 
=Pollution Index 


Page 2 


(%) Sssaquiny 4ayWoOYD 


=Summer Rainfall (mm 


2. Use these tools to investigate possible causes of the fall in chortler numbers. 


Q Write your conclusions on paper. 
Say how you would present the data to clearly support your conclusions. 


(Source: Ripley, 2009) 
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Extended responses can provide particularly useful insight into a student's level of conceptual 
understanding and reasoning. They can also be used to probe a student’s capability to analyze a 
situation and choose and carry out a plan to address that situation, as well as to interpret the stu- 
dent’s response. Students may also be given an opportunity to explain their responses, their reason- 
ing processes, or their approaches to the problem situation. They can also be asked to communicate 
about the outcomes of their approach to the situation. Care must be taken, however, particularly 
with fourth graders and English language learners, that language capability is not confounded with 


technology and engineering literacy. 


Selected Response 


As the name implies, selected-response items are those in which students read a question and 
are presented with a set of responses from which they choose the best answer. In other NAEP 
assessments, selected-response items most often take the form of multiple-choice items, in which 
students select an answer from, say, four options provided. The choices include the most applicable 
response—the “answer”—as well as three “distractors.” The distractors should appear plausible to 
students but should not be justifiable as a correct response, and, when feasible, the distractors should 
also be designed to reflect current understanding about students’ mental models in the content area. 
The NAEP Technology and Engineering Literacy Assessment will include such multiple-choice 
items within both the scenario-based and the discrete-item assessment sets as one type of selected 


response. 


In addition to the conventional multiple-choice selected-response items, the scenario-based 
sets in the NAEP Technology and Engineering Literacy Assessment will include other types of 
selected-response formats. The computer-based nature of the scenarios will allow a variety of types 
of student selections to be measured. For example, a student might be given a task to perform and 
asked to select an individual tool from a set of virtual tools. When a student selects a tool by clicking 


on it, it provides a measurable response that is, in essence, a selected response. 


A selected-response item in such a scenario might have fewer choices than in a conventional 
multiple-choice item. For example, a student might select between two alternatives, such as deciding 
whether a switch in a circuit should be open or closed in order to produce a particular outcome, 
but the student might also have to justify or explain the selection. In this case, the first part of the 
answer is a selected response, but it might be necessary to score the two parts of the item together so 
that the selection and justification together determine the score. In complex, real-world scenarios, it 
might be the case that there is not a “correct” selection, and in such a situation what matters is that 


the selection is justified adequately. 
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An example of a selected-response item that might be part of a series of items embedded in a 
scenario-based assessment set is shown on the next page. The question was part of an ICT assessment 
about reintroducing lynx into a Canadian park overrun by hares. The assessment was designed for 


13-year-olds. 


Park rangers noticed the problem because they’ve been estimating the 
number of hares in the area for the last four years. Here’s what they 
found. Last year, 2002, there were about 95,000 hares. The year before 
that, 2001, there were about 80,000. In 2000, there were 25,000. And 
in 1999, there were only about 1,000 hares. 

Your task is to organize the data to see if there is a trend. 


Pick a tool to use: 


(Source: Quellmalz & Kozma, 2004) 


Other selected-response types within a scenario-based assessment set might include a task in 
which a student selects all of the options that apply from a given set of choices. Again, in a real- 
world situation there might be one “best” combination of choices but also one or more other 
combinations that are partially correct. In such a situation, it makes sense to use a scoring rubric that 


rewards different combinations of selected-response items with different scores. 


In the following item, students observe organisms interacting in an ecosystem before choosing 
all of the organisms that are consumers. The assessment was designed for grade 8. In the NAEP 
Technology and Engineering Literacy Assessment, students might choose all of the organisms in an 


ecosystem most immediately affected by a pollutant. 
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Select all the organisms Alewife 


that are consumers. tal 
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(Source: Quellmalz, Timms, & Buckley, 2009) 


| 108 


Another form of selected response is a “hot spot” in which a student answers by clicking on a 
spot on an image such as a map, picture, or diagram. The example below was designed for grade 8 
students as part of a set of items on a state science test. For the NAEP Technology and Engineering 
Literacy Assessment, a task might be designed that asks students to click on the places in the image 


where pollution originated in the water system. 


(Source: Minnesota Department of Education, 2009) 


Ways of Measuring Student Responses 


The computer-based administration of the scenario-based assessments combined with the broad 
range of selected and constructed responses possible with this approach will provide many oppor- 
tunities to measure students’ capabilities as defined in the assessment targets. The range of measures 
will be greater than those generated in a typical NAEP assessment of other subjects, so it is necessary 
to describe how all of these measures might be handled. It is helpful to think of the measures as 
falling into two categories: student direct responses, and pattern-tracking measures that are based on 


student interactions with the tools and systems portrayed in the scenarios. 
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Conventional items always involve the student in a direct response. For example, after being pre- 
sented with information in a diagram, the student is asked a text-based question and given a limited 
set of choices from which to select the best answer. Student direct responses can also be used in 
scenarios. For example, an assessment task in the scenario may have asked the student to set two dif- 
ferent values for a component of the system and observe what happens. The student direct response 
comes when, after observing the interaction with the system, the student is asked, for example, to 
compare and contrast the two outcomes and explain in a short written response why they happened 
as they did. This is a student direct response because, although the student interacted with the 
system, none of that interaction was captured to score the appropriateness of the student response to 


the item. Only the written observation and explanation are to be scored. 


One type of student direct response is selection from a set of choices—for example, multiple 
choice, checking all of the boxes that apply, or, in a scenario-based assessment, selecting an object or 
choosing a tool for the task. Other types of direct response include providing a written analysis of a 


set of results and writing a short explanation of why a selection was made in a scenario. 


By contrast, in pattern-tracking measures the interactions that the student engaged in may pro- 
vide relevant evidence about whether the student possesses a skill that is an assessment target and 
that should, therefore, be captured, measured, and interpreted. For example, a student may have 
been asked to pinpoint a malfunction in a technological system, such as a leak in a lawn sprinkling 
system. In responding to that task, the student’s manipulation of the components of the system 
shows whether the student is testing the sprinkler components in a random or systematic way. Thus 
the things that the student chose to manipulate, how the student manipulated them, and how long 
it took might all be measured and interpreted in combination so as to provide a measure of whether 


the student possesses a particular skill related to troubleshooting. 


One type of pattern-tracking measure is the observation of patterns of action—for example, 
capturing a sequence of actions taken to determine if the correct set of actions was taken and if 
the actions were executed in the optimal order. Another pattern-tracking measure is tracking the 
manipulations that the student performs in a scenario. How, for instance, did the student change 
the features of a Web search query (for example, narrowing the topic) or vary the parameters that 
control a component of a system (for example, changing the gauge of wire mesh for bird protection 
in a model of a rooftop wind turbine) or transform an object from one form into another (for exam- 
ple, transforming database entries into a graph or table for a presentation)? Pattern-tracking mea- 
sures might be used to assess certain aspects of communication or collaboration skills. For example, 
measuring the number of times a student communicates with virtual team members with particular 


expertise can provide a measure of the efficiency of the student's collaboration strategies. 
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Balance of the Assessment 


To ensure an appropriate distribution of the test time, it is important to balance the different 
components of the assessment. This section discusses how this can be done. Note that “total test 
time” refers to the length of time that it would take to administer all available assessment blocks (see 
figure 3), which is the equivalent of approximately 8 hours of test time. Many students will be in 
the sample population during each administration of the test, but each of these students will spend 
approximately 60 minutes on the assessment tasks, so there must be a plan for distributing each 
assessment block to many students. Figure 3 also shows a simple example of how assessment types 


might be grouped together. 


ANSWER: New update based on checks. 


If all test items given to a student -> total test time of about 8 hours 


NZ 


Items are placed i test blocks of 30 mins 


amet Ei i 
a a 


ANSWER: New update based on confirmations received. 


. ETC. 


Long Scenario Short/Medium Scenario =, Discrete items 
Approx. 30 min Approx. 10-20 min ae Approx. 12 min 
Many selected Smaller number of ee Stand-alone items 
response, short selected response, any not in a scenario. 
constructed and short oath May be selected 
response, and constructed ame response or short 
long constructed response x Je ‘NJ constructed 
response ae response 


Figure 3 — Potential Balance of Total Test Time 
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This section discusses four separate aspects of the assessment that should be considered in 
determining an overall balance: 


e Balance by Major Assessment Area 
© ‘Technology and Society 
© Design and Systems 
© Information and Communication Technology 
e Balance by Practice 
© Understanding Technological Principles 
© Developing Solutions and Achieving Goals 
© Communicating and Collaborating 
e Balance by Set Type 
© Long scenarios 
© Short/medium scenarios 
o Discrete items sets 
e Balance by Response Type 
© Selected response 


© Constructed response 


The balance required at each grade level is specified in the following sections as a percentage of 
the total test time. In other words, the percentage expresses what proportion of the total amount of 
testing time—as represented in the total item pool shown at the top of the diagram in table 4.1— 
would be allocated. Since a student is assigned to take only one of the groups of item sets shown in 
table 4.1, the percentages do not necessarily represent the distribution of time in any single student’s 
test session. It should also be noted that the percentage of testing time distributions shown in tables 
4.1 through 4.4 are to be regarded as targets, and it is understood that percentages in the actual 


assessment may vary slightly from the target percentages. 


The balances shown in the following tables 4.1 through 4.4 were determined after careful con- 
sideration and deliberation by both the planning committee and the steering committee members. 
They reflect the expert judgment of the committees after they took into account how the knowledge 
and skills in the area of technology and engineering literacy are taught at different grade levels and 
how they develop over time. The percentages shown in the tables were derived from a process in 
which planning committee members made individual judgments, the group results were tabulated, 


and then discussion followed to come to a consensus that is represented in tables 4.1 through 4.4. 
Assessment Balance by Major Assessment Area 


Table 4.1 shows the balance by major assessment area at each of the three grade levels. At 


fourth grade there is an emphasis on ICT because the focus of technology and engineering literacy 


| 112 


instruction at that grade is on using common information and communication technologies. At 
eighth grade the balance is weighted to Design and Systems because in middle school there is more 
emphasis on systems, and there is slightly less time spent on ICT than in the early grades. At 12th 
grade the balance is slightly weighted to Design and Systems and to ICT. 


Table 4.1 Assessment balance by major assessment areas and grades 


Grade Grade 4 Grade 8 Grade 12 
(% total test time) (% total test time) (% total test time) 
Major 
Assessment Area 
Technology and 25 25 30 
Society 
Design and Systems 30 40 35 
Information and 
Communication 45 35 35 
Technology (ICT) 


Assessment Balance by Practice 


The balance of the assessment by crosscutting practices across the three grade levels is shown in 
table 4.2. At all grades the balance of total test time is: 


e Understanding Technological Principles - 30 percent 
e Developing Solutions and Achieving Goals — 40 percent 
e Communicating and Collaborating — 30 percent 


The rationale for a slight emphasis on the practice of Developing Solutions and Achieving Goals 


is that it is important for students to be able to use their knowledge of technological principles in 
developing solutions to problems. 
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Table 4.2 Assessment balance by practices and grades 


Assessment Balance by Set Type 


Grade Grade 4 Grade 8 Grade 12 
(% total test time) (% total test time) (% total test time) 

Practice 
Understanding 
Technological 30 30 30 
Principles 
Developing 
Solutions and 40 40 40 
Achieving Goals 
Collaborating and 

nea 30 30 30 
Communicating 


In addition to ensuring a balance across the content of the assessment, table 4.3 specifies how the 


total amount of testing time should be balanced across the three types of assessment sets—long sce- 


nario, short/medium scenario, and discrete item sets. There is an emphasis on the scenarios because 


they allow for a wide range of interactive tasks that are well-suited to assessing the types of practices 


that students need to apply in practical investigations. The 20 percent of discrete items allows for 


more independent testing of knowledge and skills to help maintain test reliability. 


Table 4.3 Assessment balance by set types and grades 


per item) 


Grade Grade 4 Grade 8 Grade 12 
(% total test time) (% total test time) (% total test time) 

Set Type 
L S i 

ong Scenarios ti i a6 
(Approx. 30 minutes) 
Short/Medium 
Scenarios 40 40 40 
(Approx. 10-20 
minutes) 
Discrete items 
(Approx. 1-2 minutes 20 20 20 
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Assessment Balance by Response Type 


Table 4.4 specifies the balance of assessment response types across the total testing time. Across 
all grade levels, there will be an emphasis on constructed-response items because the more com- 
plex tasks involved in scenarios demand that students are involved in activities that create products 
rather than simply select among choices of responses. Since the balance of assessment types specified 
in table 4.3 is weighted toward scenarios, there is a corresponding weighting toward constructed 
responses, although selected responses also form an important part of the assessment. 


Table 4.4 Assessment balance by response types and grades 


Grade Grade 4 Grade 8 Grade 12 
(% total test time) (% total test time) (% total test time) 
Response Type 
Selected Response 40 40 40 
Constructed Response 60 60 60 


Assessment Design and Student Diversity 


Students should have the opportunity to demonstrate their knowledge of the concepts and 
ideas that the NAEP Technology and Engineering Literacy Assessment is intended to measure. The 
assessment needs to be responsive to the challenges that stem from an increasingly diverse student 
population in the nation, the inclusion of all types of students in the general curriculum, and an 
increased emphasis and commitment to serve and be accountable for all students. NAEP should 
strive to develop assessments that allow for the participation of the widest possible range of students 
so that interpretation of scores of all who participate leads to valid inferences about the levels of 


their performance. 


As the NAEP Technology and Engineering Literacy Assessment will be computer-delivered 
and will contain a large proportion of scenario-based assessment items, great attention needs to be 
paid to ensuring that the assessments are valid and accessible to a wide range of students. Two types 
of populations need to be considered in order to properly address student diversity in the design 
of the assessment, English language learners—students who are developing English as their sec- 
ond language—and students with disabilities. Both populations are tremendously heterogeneous: 
the former, because of the wide range of students’ first languages, cultural influences, and stages in 
English development; the latter, because of the wide range of disabilities—from physical to sensory 


to cognitive. Whereas both populations may have common sets of needs, some design issues may 
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be particularly relevant to one or the other. Both populations should be considered carefully in the 


design of the NAEP Technology and Engineering Literacy Assessment. 


To effectively include English language learners and students with disabilities in the technology 
and engineering literacy assessments, two areas that need to be addressed include the participation 
of students in the pilot stages of test development and the use of testing accommodations. These 
aspects are discussed in more detail in the Assessment and Item Specifications for the NAEP Technol- 
ogy and Engineering Literacy Assessment. 


Pilot Student Samples 


The term “inclusion” does not refer solely to the participation of students with special needs in 
large-scale testing. It also refers to the notion that, to attain equity in testing, appropriate actions 


should be taken to ensure that these students participate in the process of assessment development. 


Issues related to students with special needs should not be addressed only at the end of the pro- 
cess of assessment development, guided by the erroneous belief that testing accommodations pro- 
vided during test administration will ensure equitable testing. Rather, English language learners and 
students with disabilities should be included in all of the assessment tryout stages in which informa- 
tion is collected from pilot students about the ways in which they interpret stimulus materials and 
test items and the difficulties they experience in providing their responses. Even when assessments 
are to be administered only in English, English language learners can provide test developers with 
valuable information for improving the assessment (for example, the wording of its items). Also, the 
information provided by students with disabilities can help to identify and address usability issues 


that are relevant to administering the assessment to all students. 


A potential challenge in the testing of students with special needs is the fact that the critical 
characteristics of these students are not always carefully identified (Solano-Flores, 2009). As a result, 
they are not properly represented in the samples of students who participate in the pilot stages of 
assessment development. The use of very small samples of English language learners and students 
with disabilities makes it difficult for test developers to properly address the heterogeneity of these 
groups. Thus, preparation for the operational assessment should address the diversity within the 
population of English language learners (for example, diverse first language backgrounds and dif- 
ferent levels of English development) and within the population of students with disabilities (for 


example, diverse types of disabilities and various levels of severity of disability). 
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Universal Design 


The items in the NAEP Technology and Engineering Literacy Assessment that make use of the 
multimedia capabilities of the computer offer a chance to more easily reach a wide range of students 
than traditional paper-and-pencil items allow. Universal design is a concept that started in architec- 
ture and has been applied to assessment by identifying the relevant essential elements for assessment 
(Thompson, Thurlow, & Malouf, 2004). Since its early explication, the concept has been applied to 
instruction as universal design for learning (Orkwis & McLane, 1998; Rose & Meyer, 2002). At the 
time of the writing of this framework, NAEP is considering how the principles of universal design 


might be applied in its assessments, but no policy has yet been adopted. 


The Universal Design for Computer-Based Testing (UD-CBT) framework and a detailed set of 
UD-CBT guidelines (Dolan et al, 2007) specifically address the design of novel computer-delivered 
assessments such as the one that will be developed for the NAEP Technology and Engineering Liter- 
acy Framework. The UD-CBT represents, at the time of publication of the NAEP Technology and 
Engineering Literacy Framework, a useful guideline for developing interactive scenario-based assess- 
ments. However, developers of the assessment will need to pay attention to developments in the 
field as a greater number of innovative assessments are developed and more information is available 


about how to apply the principles of universal design to them. 


Accommodations 


NAEP strives to assess all students selected by its sampling process. Rigorous criteria are applied 
to maximize the number of English language learners and students with disabilities included in 
NAEP assessments. Participating students with special needs are permitted to use accommodations, 


as stated in current NAEP policy: 


All special-needs students may use the same accommodations in NAEP assessments that 
they use in their usual classroom testing unless the accommodation would make it impossi- 
ble to measure the ability, skill, or proficiency being assessed, or the accommodation is not 
possible for the NAEP program to administer (NCES, 2005, Current Policy section, 4). 


To meet this commitment, it will be necessary for the assessment delivery system to include 
tool options that allow the students to benefit from the sorts of accommodations that they need. 
An example of these tool options is screen magnification, which can benefit blind or low-vision 
students. Another example is text-to-speech, which may also benefit blind or low-vision students as 
well as English language learners whose proficiency in English is better in the listening mode than in 


the reading mode. 
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To make accommodations more likely to successfully serve students with special needs, an 
important fact should be taken into account in assessment design. Each English language learner is 
unique as to his or her level of English proficiency in the listening, speaking, reading, and writing 
modes and each student with disabilities is unique as to the kind and severity of his or her disabil- 
ities. As a consequence, what works for one student does not necessarily work for another student 
within the same group of students with special needs. Designing an assessment delivery system that 
is capable of providing all NAEP-authorized accommodations and that allows selection of the set of 
accommodations that best meets the needs of each student is critical to properly meeting the goals 


of inclusion in the NAEP Technology and Engineering Literacy Assessment. 
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CHAPTER FIVE 


REPORTING RESULTS OF THE NAEP TECHNOLOGY AND ENGINEERING 
LITERACY ASSESSMENT 


Introduction 


The purpose of this chapter is to explain how the results of the NAEP Technology and Engi- 
neering Literacy Assessment will be reported. The first NAEP Technology and Engineering Literacy 
Assessment was administered in 2014 to a national sample of eighth-grade students in public and 
private schools. In 2018, the assessment will be administered again to eighth graders nationally. 

The description in this chapter applies to each administration of the assessment. Key sections of the 


chapter are as follows: 


e How NAEP Results Are Reported; 

e Reporting Scale Scores and Achievement Levels; 
e Reporting Contextual Variables; and 

e Uses of NAEP Reporting. 


How NAEP Results Are Reported 


The National Assessment of Educational Progress provides the only nationally representative 
report on student achievement in a variety of subjects. Starting in 2013, NAEP results have been 
published mainly online, on a website that provides the public with access to an interactive version 
of the report card. An executive summary will be available in print to accompany the online data. 
The online resource provides detailed information on the nature of the assessment, the demograph- 


ics of the students who participate, and the assessment results. 


The Nation’s Report Card includes information on the performance of various subgroups of 


students at the national level. Subgroups for NAEP include: 


e Gender; 

e Race/ethnicity; 

e Eligibility for free/reduced-price lunch; 
e Students with disabilities; and 


e English language learners. 
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Detailed data on NAEP results, demographic variables, and subject-specific contextual informa- 
tion are available via the NAEP Data Explorer on the website. Additional restricted data are avail- 
able for scholarly research, subject to National Center for Education Statistics (NCES) licensing 


procedures. 


The Nation’s Report Card provides results on the performance of students in public schools in 
various states as well as in the NAEP Trial Urban Districts. The Trial Urban District Assessment was 
initiated in 2002 to report on the achievement of public school students in large urban districts. The 
NAEP Technology and Engineering Literacy Assessment will not be administered as part of the 


Trial Urban District Assessment program. Results will be reported only at the national level. 


Reporting Scale Scores and Achievement Levels 


Results of the NAEP Technology and Engineering Literacy Assessment will be reported in 
terms of percentages of students who attain each of the three achievement levels, Basic, Proficient, 
and Advanced, discussed below. The NAEP Technology and Engineering Literacy Assessment is an 
assessment of overall achievement, not a tool for diagnosing the needs of individuals or groups of 
students. Reported scores are always at the aggregate level. By law and by design, scores are not pro- 
duced for individual schools or students. Results will be reported for the nation as a whole as well 
as for regions of the nation. The NAEP Technology and Engineering Literacy Assessment will not 
provide results for individual states, as the student samples will be drawn to report at the national 


level only. 


The project committees have recommended that the results of the assessment be reported in 
terms of three subscores, each of them reflecting performance in one of the three main areas of tech- 
nology and engineering literacy: Technology and Society, Design and Systems, and Information and 
Communication Technology. An overall composite score will also be reported. At this time tech- 
nology instruction in most K-12 schools focuses on one or another of the three areas rather than a 
fusion of the three, so a composite score can be expected to have less relevance than the scores from 
the three areas. Scale scores will also be available for the three practices addressed in the Technology 
and Engineering Literacy Assessment: Understanding Technological Principles; Developing Solu- 
tions and Achieving Goals; and Communicating and Collaborating. 


Reporting on achievement levels is the primary way in which NAEP results reach the general 
public and policymakers. Achievement level results indicate the degree to which student per- 
formance meets the standards set for what students should know and be able to do at the Basic, 
Proficient, and Advanced levels. Definitions of achievement levels articulate expectations of perfor- 
mance at each grade level. They are reported as percentages of students within each achievement 


level range, as well as the percentage of students at or above Basic and at or above Proficient ranges. 
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Results for students not reaching the Basic achievement level are reported as below Basic. Results are 
also reported for subgroups of students using demographic data and contextual variables specific to 
the NAEP Technology and Engineering Literacy Assessment. An individual student’s performance 
cannot be reported based on NAEP results. 


Table 5.1 displays the Governing Board’s generic policy definitions for Basic, Proficient, and 
Advanced achievement that pertain to all NAEP subjects and grades. 


Table 5.1 Generic achievement level policy definitions for NAEP 


Achievement Level Definition 


Advanced This level signifies superior performance. 


reaching this level have demonstrated competency over challenging subject matter, 


Profici 
pone including subject-matter knowledge, application of such knowledge to real-world 
situations, and analytical skills appropriate to the subject matter. 
Basi This level denotes partial mastery of prerequisite knowledge and skills that are 
asic 


This level represents solid academic performance for each grade assessed. Students 


fundamental for proficient work at each grade. 


There are three components to the NAEP achievement levels: 


e Achievement level definitions; 
e Cut scores; and 


e Examples of students’ responses. 


Achievement Level Definitions 


Since 1990, the Governing Board has used student achievement levels for reporting results on 
NAEP assessments. The achievement levels represent an informed judgment of “how good is good 
enough” in the various subjects that are assessed. Generic policy definitions for achievement at the 
Basic, Proficient, and Advanced levels describe in very general terms what students at each grade level 
should know and be able to do on the assessment. Technology and engineering literacy achieve- 
ment levels specific to the NAEP Technology and Engineering Literacy Framework are developed to 
elaborate the generic policy definitions of Basic, Proficient, and Advanced achievement for NAEP 
assessments. Using student performance results from the first assessment administered in 2014 
to eighth graders nationally, the achievement levels for the NAEP Technology and Engineering 
Literacy Assessment have been set for eighth grade in preparation for reporting the results. Prelim- 
inary achievement level definitions have been developed for each of the three areas to be reported 


separately in the assessment. They will be used to guide item development. They will also be used 
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to guide the initial stages of standard setting for the NAEP Technology and Engineering Literacy 
Assessment when the assessment is administered at grades 4 and 12. Standard setting has already 
been conducted for grade 8, and the grade 8 achievement level definitions were finalized during this 
process. (See appendix G for the preliminary achievement level definitions at grades 4, 8 and 12. See 


appendix H for the final achievement level definitions at grade 8.) 


The preliminary achievement level definitions will be revised when actual student responses 
have been collected and analyzed. The Governing Board will convene panels of experts to examine 
the preliminary achievement level definitions and to recommend final achievement level definitions 
for each grade level. A broadly representative panel of exceptional teachers, educators, and profes- 
sionals will then be convened to engage in a standard-setting process to determine the cut scores 
that correspond to these achievement level definitions. The panelists will be trained and will engage 
in a series of discussions designed to ensure informed judgments about mapping cut scores to the 


assessment. 


Cut Scores 


Cut scores, the second component of reporting on achievement levels, represent the minimum 
score required for performance at each NAEP achievement level. Cut scores are reported along with 
the percentage of students who scored at or above the cut score. As described in chapter four, the 
assessment design for the 2018 NAEP Technology and Engineering Literacy Assessment incor- 
porates scores from selected responses and written responses, as well as measures of the patterns of 
action a student takes in problem-solving. Selected responses in which there is a single best answer 
will be scored as correct or incorrect and written responses will be scored using a rubric that rewards 
answers according to their match to descriptions in the rubric. The pattern-tracking will be evalu- 
ated by comparing the pattern of action against a set of possible patterns, and students will get more 
credit for a course of action that is optimal than for alternative patterns of action. Scores can then be 


combined to produce overall scores so that cut score decisions can be made. 


Examples of Students’ Responses 


The third component of achievement level reporting includes examples of student responses on 
released tasks from the NAEP Technology and Engineering Literacy Assessment. These examples 
provide illustrations of student skills within each level of achievement for each of the three areas and 
will be developed after the first administration of the assessment. Example responses will be anno- 


tated to explain the score for the response. 
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Reporting Contextual Variables 


Contextual data on students, teachers, and schools are needed to fulfill the statutory require- 
ment that NAEP include information, whenever feasible, for groups identified in the first section of 
this chapter (for example, gender, race/ethnicity). Therefore, students, teachers, and school admin- 
istrators participating in NAEP are asked to respond to questionnaires designed to gather demo- 
graphic information. Information is also gathered from non-NAEP sources, such as state, district, or 
school records. For the 2018 NAEP Technology and Engineering Literacy Assessment, only student 
and school information will be collected, since many students will not have taken a separate course 


in technology and engineering literacy taught by a specific teacher. 


In addition to demographic information, contextual questionnaires include questions about 
variables related to opportunities to learn and achievement in technology and engineering literacy. 
The variables are selected to be of topical interest, to be timely, and to be directly related to academic 
achievement and current trends and issues in technology and engineering literacy. Questions do 
not solicit information about personal topics or information irrelevant to the collection of data on 


technology and engineering literacy achievement. 
The important components of NAEP reporting are summarized in table 5.2. Recommendations 
for contextual variables for the 2018 NAEP Technology and Engineering Literacy Assessment are 


presented in the separate contextual variables document. 


Table 5.2 Summary of NAEP reporting components 


Key Characteristics 


Starting in 2013, elements released to the public will include: 


How e Results published mainly online with an interactive report card 
Information e Dedicated website: www.nationsreportcard.gov 
Is Reported e Performance of various subgroups at the national level, published in print 
and online 


NAEP data are reported by: 

What Is e Percentage of students attaining achievement levels 
Reported e Scale scores 
e Sample responses to illustrate achievement level definitions 


Components of NAEP Reporting 


What 
. Types of contextual variables distributed to students and schools: 
Information . f 
e These are presented in the separate contextual variables document. 
Is Gathered 
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Uses of NAEP Reporting 
The information available from results of the 2018 NAEP Technology and Engineering Literacy 


Assessment will provide important data that can be used throughout the tenure of the framework. 
The results of the assessment will continue the trend line from the 2014 assessment, and policy- 
makers, educators, and the public can use data from the assessments as a tool for monitoring certain 
aspects of student achievement in technology and engineering literacy over time. NAEP reports 
from any subsequent administrations of the assessment will compare student performance in the 
three areas of Technology and Society, Design and Systems, and ICT among groups of students 
within the same grade. Long-term achievement trends (for example, the comparison of score perfor- 


mance with previous administrations) can also be reported starting with the second administration. 


The scores from the assessment will be of value and interest not just to technology and engineer- 
ing teachers but to a broad range of educators. As discussed earlier, many different types of teachers 
are involved in teaching their students about technology and its applications in grades K-12, from 
those specializing in science, math, and engineering, to those teaching social sciences, humanities, 


and the arts as well as members of cross-disciplinary teams. 


Because the NAEP Technology and Engineering Literacy Assessment will measure some tech- 
nology and engineering literacy experiences, but not all, there will be limitations to the range and 
scope of information it can produce. NAEP publishes data on student performance in relation to 
various achievement levels and demographic subgroups; the information reported does not evaluate 
results or provide conclusive statements about the level of achievement among the nation’s K-12 
students. Furthermore, the NAEP Technology and Engineering Literacy Assessment is not designed 
to inform instruction—to guide how technology and engineering literacy is taught—but only to 
measure the performance of a representative sample of the American student population at the des- 


ignated grade within the assessment context outlined in this framework. 
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APPENDIXES 


APPENDIX A: GLOSSARY OF ABBREVIATIONS, WORDS, AND TERMS USED 
IN THE FRAMEWORK 


The glossary is divided into five sections. The first section presents acronyms and abbreviations 
used in the framework, followed by sections on basic framework terminology, assessment terms, 
and terms related to education content and pedagogy. The final section presents terms specific to 
the three major assessment areas of technology and engineering literacy: Technology and Society, 
Design and Systems, and Information and Communication Technology. Relevant terms and defini- 


tions from each area are included, and they are defined within the context of the framework. 


Acronyms and Abbreviations for Associations, Educational Organizations, or Reports 


AAAS American Association for the Advancement of Science 
CCSSO Council of Chief State School Officers 

IBO International Baccalaureate Organization 

ISTE International Society for Technology in Education 

ITEEA International Technology and Engineering Educators Association 
NAE National Academy of Engineering 

NAEP National Assessment of Educational Progress 

NCES National Center for Education Statistics 

NETS:S National Educational Technology Standards for Students 
NRC National Research Council 

NSTA National Science Teachers Association 

OECD Organisation for Economic Co-operation and Development 
PISA Programme for International Student Assessment 

SETDA State Educational Technology Directors Association 

STEM Science, Technology, Engineering, and Mathematics 

TIMSS Trends in International Mathematics and Science Study 
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Basic Framework Terminology 


Technology 


Engineering 


Technology and engineering 


literacy 


Educational technology 


Technology educators 


Technology education 


Framework 


Te chnological processes 


Technological principles 


Technological practices 
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Any modification of the natural or designed world done to fulfill 


human needs or desires. 


A systematic and often iterative approach to designing objects, pro- 


cesses, and systems to meet human needs and wants. 


Capacity to use, understand, and evaluate technology as well as 

to understand technological principles and strategies needed to 
develop solutions and achieve goals. It encompasses the three areas 
of Technology and Society, Design and Systems, and Information 


and Communication Technology. 


Various types of equipment, tools, and processes used as aids in 


teaching and learning. 


All those whose teaching responsibilities include imparting the 


knowledge, capabilities, and skills described in this framework. 


The knowledge and skills taught to students in the three areas of 
Technology and Society, Design and Systems, and Information and 


Communication Technology. 


A NAEP framework is a document that defines the parameters 
of a NAEP assessment. It guides the test makers in developing an 


assessment. 


Procedures using technological knowledge, tools, and skills to 


develop solutions and achieve goals. 


Sets of foundational and fundamental assumptions that under- 
lie each of the three areas of technology and engineering literacy 


defined in this framework. 


Types of thinking and reasoning that students are expected to 
demonstrate when responding to an assessment item. The frame- 
work specifies three practices: understanding technological princi- 
ples; developing solutions and achieving goals; and communicating 


and collaboration. 


Assessment Terms 


Advanced achievement level 


Assessment areas or targets 


Assessment balance 


Assessment specifications 


Contextual variables 


Basic achievement level 


Constructed response 


Cut scores 


Discrete item set 


Item 


Item specifications 


Probe (noun) 


The highest of NAEP’s three levels of performance. This level signi- 


fies superior performance. 


The three assessment foci of this framework on technological liter- 
ary: Technology and Society, Design and Systems, and Information 


and Communication Technology. 


Appropriate distribution of items according to major assessment 


area, technological practice, assessment set type, and response type. 


Assessment requirements that framework developers give to test 
developers. These include, for example, the foci of the assessment, 
the number and types of items, the specific areas to be assessed, the 


accommodations for students with disabilities, etc. 


Demographic and contextual data related to the NAEP assessment 
gathered through questionnaires usually completed by school 


administrators, teachers, and students. 


The lowest of NAEP’s three levels of performance. This level 
denotes partial mastery of prerequisite knowledge and skills that are 


fundamental for proficient work at each grade. 


Items in which the student “constructs” the response rather than 
choosing a response from a limited number of alternatives. Con- 
structed responses may be short (students supply a word or short 
sentence) or extended (students complete a task). 


The minimum score required for performance at each NAEP 


achievement level. 


A group of questions that include conventional selected-response 


items and short constructed-response items. 
A single question or set of instructions. 


Assessment requirements that framework developers give to test 


developers; for example, the number and types of items to be 


included. 


A smaller-scale, focused assessment on a timely topic that explores a 


particular question or issue and may be limited to particular grades. 


(continued) 
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Assessment Terms (continued) 


Proficient achievement level 


Response type 


Scale scores 


Scenario -based assessment 


Selected response 


Universal design 


for assessment 


The middle of NAEP’s three levels of performance. This level 
represents solid academic performance for each grade assessed. 
Students reaching this level have demonstrated competency over 
challenging subject matter, including subject matter knowledge, 
application of such knowledge to practical situations, and analytical 


skills appropriate to the subject matter. 


The activity an item asks a student to perform when responding. 
In this assessment there are three item response types: short con- 


structed response, long constructed response, and selected response. 


Scores that allow for comparison of students’ performances on dif- 
ferent administrations of a test. For example, students’ scores might 


be converted to a score on a scale that ranges from 0 to 300 points. 


In the context of this framework, scenarios are interactive computer 
tasks that constitute the bulk of the items. Scenarios may be short or 


long, depending on what they require the student to do. 


A type of item in which students read a question and choose the 


best answer from a set of options. 


Guidelines for ensuring that the largest number of disabled students 


and English language learners participate in an assessment. 


Education Content and Pedagogy 


Academic problem 


Collaboration 


Diffuse curriculum 


Disaggregation 
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An assigned task that a teacher may give to a student. 


To work together with other individuals. However, for this NAEP 
assessment, it will mean using contemporary technologies to work 
with virtual (computer-generated) individuals to solve problems or 


achieve goals. 
Curriculum without a clear scope, sequence, and series of courses. 


Separation into component parts, such as the breaking down of 


achievement data by racial/ethnic subgroup. 


Equity 


Fluency 
Habits of mind 


Literacy 


Sequential curriculum 


Area-Specific Terms 


Technology and Society 


Artifacts 


Human-made 


Modeling and simulation 


Natural world 


Practical problem 


Product life cycle 


Fair access to opportunities to learn that are based on need rather 


than on some arbitrary factor. 
A smooth and easy flow of knowledge and skills. 
Customary ways of thinking and acting. 


The capacity to use, understand, and evaluate a body of knowledge 
and skills as well as to apply concepts and processes to solve prob- 


lems and reach one’s goals. 


Curriculum that has a scope, sequence, and a series of courses. 


Products and items that a society’s population develops, uses, and 


updates to meet needs and wants. 


Term describing an artifact that has been designed and developed by 
means that are outside the boundaries and capabilities of the natural 


world. 


Utilizing technology to analyze and test the possible effects, 
impacts, and trade-offs that are associated with a new technological 
innovation to evaluate efficiency, discover potential problems, and 


develop workable solutions. 


Plants, animals, water, and other organisms and elements that exist 


without contributions from humans. 


A situation that a person may encounter in everyday life that 


requires a solution. 


The span of time that an artifact is commissioned to satisfy a socie- 
tal need that can start from the point of design, manipulation of raw 
materials, and manufacturing processes, to eventual obsolescence 


and disposal. 


(continued) 
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‘Technology and Society (continued) 


Regulating technologies 


Technological inequalities 


Trade-off 


Design and Systems 


Constraint 


Criteria 


Engineering design method 
(sometimes called 
technological design) 

Life cycle 

Optimization 
Problem-solving 


Prototype 


Requirements 


Reverse engineering 
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Technological innovations that are responsible for contributing to 
the protection of natural resources in areas such as transportation, 


energy, and waste disposal. 


Instances where countries and societies use antiquated technologies 


due to economic circumstances or cultural preferences. 


A decision where complete awareness of both the advantages and 
disadvantages of the result are explored and the impacts of both are 


taken into consideration. 


A boundary, limit, or restriction, such as time, money, or resources, 


in the requirements for a project. 


Characteristics (or specifications) of a successful solution, such as a 


desired function or a particular level of efficiency. 


An iterative, systematic process for solving problems that involves 


creativity, experience, and accumulated disciplinary knowledge. 


Important phases in the development of a system from initial con- 


cept through design, testing, use, and maintenance to retirement. 


Finding the best possible solution when some criterion or constraint 


is identified as the most important and others are given less weight. 


The cognitive process of finding answers to questions and solutions 


to undesired situations. 


First version, or generation, of an entity created from a particular 


design plan using the engineering design method. 
Combination of criteria and constraints for a given project. 


Disassembling an item in a systematic way to understand how it 


works, usually so it can be repaired, copied, or improved. 


Systems thinking 


Troubleshooting 


Way of investigating or thinking about a system using a set of 


principles. 


Systematic method of dealing with failures. 


Information and Communication Technology 


Digital assistants 


Digital models 
Digital tools 


Fair use 


Geographical information 


system 


Information and 
Communication 


Technologies 


Information (or digital) 


literacy 


Interactive whiteboards 


Media literacy 


Also called personal digital assistants (PDAs), devices used as orga- 
nizers to enter and store data such as addresses, expenses, or calen- 
dar items. Some are capable of being used as hand-held computers 


and may also have Internet capabilities. 
An electronic representation of a system. 
Any technology that stores and transmits data electronically. 


A condition under US. copyright law that permits limited use of 


copyrighted material without procuring permission from the copy- 


right holder. 


Any system that gathers, saves, evaluates, and presents data related 
to geographic locations. A common example is a GPS, used to 


obtain driving directions. 


Technologies used to access, gather, store, analyze, and report 


information. 


Skills needed to access, evaluate, and use information from a variety 


of sources. 


An interactive display system that connects to a projector and 
a computer. Using special software, it is possible to project the 
computer’s desktop and then control the computer using a stylus, 


personal response system, or even a finger. 


The capacity to access and evaluate messages created using a variety 
of media, such as advertisements, commercials, or speeches, etc. 
It also refers to the skills required to develop and communicate a 


message using media. 


(continued) 
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Information and Communication Technology (continued) 


Media player Software used on a computer to manage and play video or audio 
files or to view digital images. It can also be hand-held hardware that 


provides the same functions and is also used to store files. 


Mobile wireless devices Small, lightweight hardware, often called hand-held technology, 
that has the capability to connect wirelessly to the Internet. Exam- 


ples include hand-held computers, smartphones, and netbooks. 


Wiki A website that allows users to work collaboratively to view, edit, and 
add information. One of the best-known examples of this type of 


site is Wikipedia, a collaboratively written encyclopedia. 
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APPENDIX B: STEERING COMMITTEE GUIDELINES* 


December 18, 2008 
Provisional Approval: September 16, 2009 


Introduction 


The steering committee guidelines that follow are framed, in part, by the Issues and Recommen- 
dations paper prepared by Sharif Shakrani (Michigan State University) and Greg Pearson (National 
Academy of Engineering) for the National Assessment Governing Board. The guidelines were devel- 
oped over a two-day period (December 17-18, 2008) by the steering committee for the 2012 NAEP 
Technological Literacy Assessment Project. The steering committee presented these guidelines to 
the planning committee in a joint session on December 18, 2008. The guidelines were revised and 
reorganized according to decisions made during the steering committee’s meeting on March 11-12, 
2009, and again shared with the planning committee. In addition, the steering committee sought 
input from leaders in education, industry, business, engineering, research, and allied organizations 
(including the International Society for Technology in Education [ISTE], the International Tech- 
nology and Engineering Educators Association [ITEEA], and the Partnership for 21st Century 
Skills); feedback resulting from these reviews served to further refine the guidelines. 


Suggested Definition for Technological Literacy 


The steering committee has identified various elements that illustrate the knowledge, ways of 
thinking and acting, and capabilities that define technological literacy. Technological literacy, as 
viewed by the steering committee, is the capacity to use, understand, and evaluate technology as well 
as to understand technological principles and strategies needed to develop solutions and achieve 
goals. It encompasses the three areas of Technology and Society, Design and Systems, and Informa- 


tion and Communication Technology. 


* These guidelines reflect the recommendations of the steering committee as of September 2009 and do not include the change in title and year of the 
assessment approved by the National Assessment Governing Board in March, 2010. Also, the initial assessment was originally slated to be conducted 
in 2012, but was administered in 2014 due to scheduling issues. 
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Recommended Grade Level to Be Assessed 


The National Assessment Governing Board requested that the two committees recommend to 
the Board at what grade(s) the national probe should be conducted in 2012. Hence, the steering and 
planning committees of the NAEP Technological Literacy Framework Development Project, in a 
joint session on March 11, 2009, discussed the matter. On the basis of this discussion, the two com- 
mittees recommend to the Governing Board that the proposed 2012 NAEP Technological Literacy 
probe be administered at grade 8. If funding is available for an additional probe, the committees 
recommend that it be given at grade 12. The lowest priority would be for a probe at grade 4. The 


rationale for choosing grade 8 included: 


e Students are cognitively mature and are more likely to have taken a technology-related 
course or curriculum units. 

e This is the last grade before student dropout rates increase. 

e Grade 8 is aligned with the Trends in International Mathematics and Science Study 
(TIMSS) and the Programme for International Student Assessment (PISA) assessments, 
allowing for more data analysis. 

e Differences in performance by gender are less pronounced than at grade 12. 

e There is more opportunity to measure impact from schooling than at grade 4. 

e Students may be better equipped to take a computer-based test than at grade 4. 

e Grade 8 is specifically targeted in No Child Left Behind, which requires that every student 
is technologically literate by the time the student finishes the eighth grade. 


Guiding Principles 


The assessment shall consist of technological con- 
tent areas making up the scale scores to be reported 
and technological practices that characterize the 


field. 


1. There are two content areas that must be assessed 
and where the data must be reported out as 


subscales: 


a. Design and Systems 
b. Information and Communication Technology 


(ICT) 


Figure B-1: The relationship between major 
assessment content areas (blue) and 
technological practices (yellow). 
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If possible, the following area should be explored as a third content area of focus: 


c. Technology and Society 


See the addendum for examples of targets to be included in the three content areas. 


Suggested technological practices are listed below: 


a. Understanding Technological Principles 
b. Developing Solutions and Achieving Goals 


c. Communicating and Collaborating 


The relationships of these practices to the content areas and to each other are reflected in figure B-1. 


Content and context for the assessment should be informed by existing state standards and 
assessments, national (for example, International Society for Technology in Education [ISTE], 
International Technology and Engineering Educators Association [[TEEA]) and international 
standards, and research (National Academy of Engineering, National Research Council). Exam- 
ples from industry, federal agencies responsible for carrying out STEM-based research (NASA, 
National Oceanic and Atmospheric Administration [NOAA], U.S. Department of Agriculture, 
National Science Foundation [NSF], U.S. Department of Energy, U.S. Department of Transpor- 
tation, etc.), and science and technology museums might also be collected for building assess- 


ment ideas. 


The assessment should have tasks that are applied to real-world contexts and should be scaf- 
folded to ensure that reliance on students’ prior knowledge of specific technology systems 
will be minimized. The focus should be on concepts and not on specific vocational skills or 


technologies. 


“Life situations” and local and contemporaneous conditions should be used as a way to confer 
relevance to each grade level. Content and context of the assessment items should have relevance 
and meaning to the learner. If possible, options should be generated in the assessment represent- 


ing different situations to remove bias due to the context of the student. 


The planning committee must develop examples of items and tasks that measure only knowl- 
edge and practices that are assessable. The examples should not be limited to multiple choice but 


should illustrate, if possible, different styles and technological tools. 


Considerable work has already been put into the development of state standards for technology, 
ICT fluency, and engineering. These existing state standards and assessments should inform the 
development of the NAEP framework, but should not limit the framework. The planning com- 
mittee should consider differences in how technological literacy is defined and treated in state 
standards, whether as discrete standards or as part of core content standards, e.g. mathematics, 


science, and language arts. Attention should also be paid to the confusion around Educational 
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Technology versus Technological Education. There are numerous studies that specify the nature 
of technological literacy within the standards of various states. States each have their own defini- 


tion of technological literacy, which may vary from the definition used in the NAEP framework. 


To avoid the potential problem of obsolescence, the 2012 Technological Literacy Assessment 
should focus on a broad base of knowledge and skills, not on specific technologies that may 

change. For example, specific communication technologies in use today (Internet-connected 
multimedia, smartphones and PDAs) would not have been familiar to students a decade ago, 


and will most likely be obsolete a decade from now. 


The assessment should use innovative computer-based assessment strategies that are informed 
by research on learning and are related to the assessment target (for example, problem-solving). 
These strategies should also reflect existing state computer-based assessments and the comput- 
er-based assessment aspects of the 2009 NAEP Science Framework. To effectively integrate inno- 
vative computer-based assessment strategies in the NAEP Technological Literacy Framework, the 
planning committee and/or assessment developers need to know the affordances and con- 
straints of various technologies, how particular ones could support assessment goals, and how to 
use them. Specific examples of tools available for assessment tasks include probeware, presenta- 


tion software, authoring tools, electronic whiteboards, drawing software, and typing tutors. 


Computer-based assessment strategies should be informed by what is known about all learners, 
including those with special needs (English language learners and students with disabilities). 
These strategies should also be informed by the ways students currently use computers to access, 
process, and utilize information (Web 2.0, social networking, etc.). Effective assessment design 
incorporates current understandings of how people learn, how experts organize information, 
and the skills of effective learners. Development of the framework and item specifications 
should draw on expertise in engineering and technology to suggest types of problems that might 
be adapted to computer-based testing. The kinds of technological tools scientists and engineers 
use, such as simulations and visualizations, should be considered for use in the technological 


literacy assessment. 


Addendum 


The steering committee formulated the following list of targets they thought should be included 
in each of the content areas. Although they consider these sets of targets to be necessary, they agree 
that the specified targets may not be sufficient. In addition, the committee recognized that guide- 
lines are normally “big ideas” and that this list contains detail in some areas and not in others. In the 
areas where detail appears, the committee considers these details sufficiently important to enumerate 


them for the planning committee’s consideration on individual targets. 


Major Assessment Areas Targets 


Technology and Society a. Where does technology come from? 
b. Why is it created? 


How is society affected by scientific and technological concepts and 


a 


capabilities? 

d. How technology affects the environment and vice versa 

e. Assessing consequences of technology (intended and unintended) 
f. Green technology 

g. Digital citizenship 

h. Safety 

i. Ethics 

j. Responsible use of technology 


Design and Systems a. Artifacts are the products of technological systems that meet a need or solve a 
problem 

Identifying the technological dimensions of an artifact 

Constraints 


Models (and their importance and limitations) 


b 

Cc 

d. 

e. Evaluation 
f. Efficiency 

g. Ethics 

h. Economy 

i. Trade-offs/cost, benefit 


Consequences 


J. 
k. Systems thinking 


Information and Research and use of information 


a 
Communication Technology b. Recognizing bias 
(ICT) c. Extracting critical information 
d. Media literacy 
e. Use of digital information systems for innovation and creative expression 
f.  Information/communication operations and concepts 
g. Use of ICT and learning technologies to learn new content and create knowl- 
edge (such as wikis, blogs, probeware, and other collaboration technologies) 


h. Data-driven decision-making 


i. Global awareness 


j. Ethics 
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APPENDIX C: ALIGNMENT TABLE — COMPARING THE NAEP TECHNOLOGY 
AND ENGINEERING LITERACY ASSESSMENT AREAS TO U.S. SOURCE 


DOCUMENTS 


2014 - 2018 NAEP - 
Technology and 
Society 


Interaction of 


ISTE — National Education 
Technology Standards 
(2007) 


Digital Citizenship: Students understand 


ITEEA - Standards for 
Technological Literacy 


(2007) 


Standard 4: Students will develop an 


5.b. Students exhibit a positive attitude 
toward using technology that supports 


collaboration, learning, and productivity. 


Technology and human, cultural, and societal issues related | understanding of the cultural, social, eco- 
Humans to technology, and practice legal and ethical | nomic, and political effects of technology. 
behavior Standard 6: Students will develop an 
5.d. Students exhibit leadership for digital | understanding of the role of society in the 
citizenship. development and use of technology. 
Standard 7: Students will develop an 
understanding of the influence of technol- 
ogy on history. 
Standard 13: Students will develop abil- 
ities to assess the impact of products and 
systems. 
Effects of Technology on Standard 5: Students will develop an 
the Natural World understanding of the effects of technology 
on the environment. 
Standard 13: Students will develop abil- 
ities to assess the impact of products and 
systems. 
Effects of Technology 5. Digital Citizenship: Standard 13: Students will develop abil- 
on the World of 5.a. Students advocate and practice safe, ities to assess the impact of products and 
Information and legal, and responsible use of information systems. 
Knowledge and technology. 


Ethics, Equity, and 
Responsibility 


5. Digital Citizenship: 
5.a. Students advocate and practice safe, 
legal, and responsible use of information 


and technology. 
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2014 - 2018 NAEP - 


ISTE — National Education 


ITEEA - Standards for 


1.a. Students apply existing knowledge to 
generate new ideas, products, or processes. 
6. Technology Operations and Concepts: 
6.a. Students understand and use technol- 
ogy systems. 


Design and Systems Technology Standards Technological Literacy 
(2007) (2007) 
Nature of Technology 1. Creativity and Innovation: Standard 1: Students will develop an 


understanding of the characteristics and 
scope of technology. 

Standard 2: Students will develop an 
understanding of the core concepts of 
technology. 

Standard 3: Students will develop an 
understanding of the relationships among 
technologies and the connections between 


technology and other fields of study. 


Engineering Design 


1. Creativity and Innovation: 

1.a. Students apply existing knowledge to 
generate new ideas, products, or processes. 
l.c. Students use models and simulations to 
explore complex systems and issues. 

3. Research and Information Fluency: 
3.a. Students plan strategies to guide 
inquiry. 

3.d. Students process data and report 
results. 

4. Critical Thinking, Problem-Solving, 
and Decision-Making: 

Students use critical-thinking skills to plan 
and conduct research, manage projects, 
solve problems, and make informed deci- 
sions using appropriate digital tools and 


resources. 


Standard 8: Students will develop an 
understanding of the attributes of design. 
Standard 9: Students will develop an 
understanding of engineering design. 
Standard 11: Students will develop abilities 
to apply the design process. 


Systems Thinking 


1. Creativity and Innovation: 

1.a. Students apply existing knowledge to 
generate new ideas, products, or processes. 
6. Technology Operations and Concepts: 
6.a. Students understand and use technol- 
ogy systems. 


Standard 13: Students will develop abil- 
ities to assess the impact of products and 


systems. 


Maintenance and 
Troubleshooting 
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6. Technology Operations and Concepts: 
6.c. Students troubleshoot systems and 


applications. 


Standard 10: Students will develop an 
understanding of the role of troubleshoot- 
ing, research and development, invention 
and innovation, and experimentation in 
problem-solving. 

Standard 12: Students will develop abilities 


to use and maintain technological products 


and systems. 


2014 —- 2018 NAEP - 
Information and 
Communication 


Technology (ICT) 


Construction and 


Exchange of Ideas and 


Solutions 


ISTE - National Education 
Technology Standards 
(2007) 


2. Communication and Collaboration: 
Students use digital media and environ- 
ments to communicate and work collabo- 
ratively, including at a distance, to support 
individual learning and contribute to the 


learning of others. 


ITEEA - Standards for 
Technological Literacy 
(2007) 


Information Research 


3. Research and Information Fluency: 
Students apply digital tools to gather, evalu- 


ate, and use information. 


Investigation of 


4. Critical Thinking, Problem-Solving, 


Standard 8: Students will develop an 


Problems and Decision-Making: Students use understanding of the attributes of design. 
critical-thinking skills to plan and conduct | Standard 9: Students will develop an 
research, manage projects, solve problems, understanding of engineering design. 
and make informed decisions using appro- | Standard 11: Students will develop abili- 
priate digital tools and resources. ties to apply the design process. 

Acknowledgment of 5. Digital Citizenship: Students under- 


Ideas and Information 


stand human, cultural, and societal issues 
related to technology and practice legal and 
ethical behavior. 


Selection and Use of 
Digital Tools 


6. Technology Operations and Concepts: 
Students demonstrate a sound understand- 
ing of technology concepts, systems, and 


operations. 


Standard 17: Students will develop an 
understanding of and be able to select 
and use information and communication 


technologies. 
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APPENDIX D: ALIGNMENT TABLE — COMPARING THE NAEP TECHNOLOGY 
AND ENGINEERING LITERACY ASSESSMENT AREAS TO INTERNATIONAL 
SOURCE DOCUMENTS 


2014 - 2018 
NAEP 


Technology and 
Society 


International Baccalaureate 


Organization Middle Years 


Programme Technology (2006) 


EU Key 
Competences for Lifelong 
Learning — A European 
Framework 
(2007) 


UK PISA 
Framework Framework 
for for 


Secondary | Assessment of 
ICT ICT Literacy 
(2008) (2006) 


Interaction of 


Students identify the problem to be 


Understanding the potential 


Information and 


Knowledge 


Ethics, Equity, and 
Responsibility 


During the course, students should: 


Carry out units of work in 
technology using materials 

and techniques safely and 
responsibly 

Provide evidence of personal 
engagement with the subject 
(motivation, independence, 
general positive attitude) when 


working in technology. 


Technology and solved. of IST to support creativity 

Humans and innovation for personal 
fulfillment, social inclusion 
and employability. 
Interest in using IST to 
broaden horizons by taking 
part in communities and 
networks for cultural, social, 
and professional purposes. 

Effects of Students identify the problem to be 

Technology on the | solved. 

Natural World 

Effects of 

Technology on 

the World of 


Positive attitude and sensi- 
tivity to safe and responsible 
use of the Internet, including 
privacy issues and cultural 


differences. 
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International EU Key 
PISA F; A 
2014 - 2018 NAEP Baccalaureate Competences for Se ence 
6 ‘ati Lifelone L ‘ UK Framework for for 
coe rate Spee ramen Secondary ICT Assessment of ICT 
Middle Years A European : 
(2008) Literacy 
Design and Systems Programme Framework (2006) 
Technology (2006) (2007) 
Nature of 
Technology 
Engineering Design | Students formulate a 2 Developing ideas 
design specification. 2.2 Models and 
Students design the modeling 
4 Evaluating 


product/solution. 


Students plan the 
product/solution. 


Students follow the 
plan. 


Students create the 


product/solution. 


4.1 Evaluating 


work 


Maintenance and 


Troubleshooting 


Systems Thinking 


1144 


Seen Ane International EU Key RISA aoe 
Baccalaureate Competences for 
Organization Lifelong Learning - EC Penman creas fof 
: Secondary ICT Assessment of ICT 
Tnconiariontand Middle Years A European , 
(2008) Literacy 
(CGommaunicition Programme Framework (2006) 
Technology (ICT) | Technology (2006) (2007) 
Construction and During the course, Propensity to use 3 Communicating Processes 
Exchange of Ideas students should IST to work autono- | information - communicating 
and Solutions work effectively mously and in teams; 3.1 Fitness for with others 
as members of a critical and reflective purpose 
team, collaborating, | attitude in the assess- 3.2 Refining and 
acknowledging, and | ment of available presenting 
supporting the views | information. information 
of others. 3.3 Communicating 
Ability to use appro- 
priate aids (pre- 
sentations, graphs, 
charts, maps) to 
produce, present, or 
understand complex 
information. 
Information Students should Ability to search, col- | 1 Finding Processes 
Research develop the design lect, and process elec- | information - accessing 
brief. tronic information, 1.1 Using data and information 
data, and concepts information - managing 
and to use them in a sources information 
systematic way. 1.2 Searching and - integrating 
selecting information 
Ability to access and 1.3 Organizing and - evaluating 
search a website and investigating information 
to use Internet-based - constructing new 
services such as dis- knowledge 
cussion forums and 
email. 
Investigation of Ability to use IST 2 Developing ideas Context 
Problems to support critical 2.1 Analyzing and - ICT for personal 
thinking, creativity, automating use 
and innovation in processes - ICT for public 
different contexts at 2.2. Models and use 
home, leisure, and modeling -ICT for educa- 
work. 2.3 Sequencing tional use 
instructions - ICT for occupa- 
tional purposes 
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tion storage, and 
management. 
Awareness of the 
opportunities given 
by the use of the 
Internet and commu- 
nication via elec- 
tronic media (email, 
videoconferencing, 
other network tools); 
and the differences 
between the real and 


virtual world. 


reer International EU Key RISA eC Ore 
Baccalaureate Competences for 
Organization Lifelong Learning - UN Penman creas fof 
: Secondary ICT Assessment of ICT 
nconiiaciontand Middle Years A European , 
(2008) Literacy 
(Commauniciton Programme Framework (2006) 
Technology (ICT) | Technology (2006) (2007) 
Acknowledg- During the course, Basic understanding 
ment of Ideas and students should of the reliability 
Information work effectively and validity of the 
as members of a information available 
team, collaborating, (accessibility/accept- 
acknowledging, and | ability) and aware- 
supporting the views | ness of the need to 
of others. respect ethical princi- 
ples in the interactive 
use of IST. 
Selection and Use of Understanding the Technology 
Digital Tools main computer Environments 
applications, includ- - Web 
ing word process- - desktop 
ing, spreadsheets, - e-learning 
databases, informa- environments 
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APPENDIX E: ALIGNMENT TABLE — COMPARING ICT SUBAREAS TO THE 
ISTE NETS:S AND THE FRAMEWORK FOR 21ST CENTURY LEARNING 


A. Construction and Exchange of Ideas and Solutions 


from a variety of sources. 


Fourth-grade students should be able to collaborate and communicate by working with other members of a (virtual) 
team to make decisions and develop presentations using a variety of formats. Eighth-grade students should be able to 
take into account the perspectives of different audiences, use a variety of media to create effective messages, and mod- 
ify presentations based on feedback (virtual). Twelfth-grade students should have developed strategies to be effective 
collaborators, should be able to take into account multiple viewpoints, and should be able to synthesize information 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

1.4.1: People collaborating as a team 
can often produce a better product 
than people working alone. There 
are common digital tools that can be 
used to facilitate virtual or face-to- 


face collaboration. 


Students know that: 

1.8.1: Collaboration can take many 
forms. Pairs or teams of people can 
work together in the same space or 
at a distance, at the same time or at 
different times, and on creative proj- 
ects or on technical tasks. Different 
communications technologies are 
used to support these different forms 
of collaboration. 


Students know that: 

1.12.1: Effective collaboration 
requires careful selection of team 
members, monitoring of progress, 
strategies for reaching agreement 
when there are opposing points of 
view, and iterative improvement of 
collaborative processes. Information 
and communication technologies can 
be used to record and share different 
viewpoints and to collect and tabu- 


late the views of groups of people. 


Students are able to: 

1.4.2: Utilize input from (virtual, 
that is, computer-generated) collab- 
orators and experts or sources in the 
decision-making process to design a 


product or presentation. 


Students are able to: 

1.8.2: Provide feedback to a (virtual) 
collaborator on a product or presen- 

tation, taking into account the other 

person’s goals and using constructive, 


rather than negative, criticism. 


Students are able to: 

1.12.2: Work through a simulation 
of a collaborative process. Negotiate 
team roles and resources, draw on 
the expertise and strengths of other 
team members and remote experts, 
monitor progress toward goals, and 
reflect on and refine team processes 


for achieving goals. 


1.4.3: Communicate information 
and ideas effectively to an audience 
in order to accomplish a specified 


purpose. 


1.8.3: Communicate information 
and ideas effectively using a variety 
of media, genres, and formats for 
multiple purposes and a variety of 


audiences. 


1.12.3: Synthesize input from multi- 
ple sources to communicate ideas to 
a variety of audiences using various 


media, genres, and formats. 
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b. 


is) 


d. 


NETS¢&S Category 2: Communication and Collaboration 


Students use digital media and environments to communicate and work collaboratively, including at a distance, to 
support individual learning and contribute to the learning of others. Students: 
a. 


Interact, collaborate, and publish with peers, experts, or others employing a variety of digital environments 
and media. 

Communicate information and ideas effectively to multiple audiences using a variety of media and formats. 
Develop cultural understanding and global awareness by engaging with learners of other cultures. 


Contribute to project teams to produce original works or solve problems. 


Framework for 21st Century Learning: Communication and Collaboration 


Communicate Clearly 


Articulate thoughts and ideas effectively using oral, written, and nonverbal communication skills in a variety of 
forms and contexts. 

Listen effectively to decipher meaning, including knowledge, values, attitudes, and intentions. 

Use communication for a range of purposes (for example, to inform, instruct, motivate, and persuade). 

Utilize multiple media and technologies, and know how to judge their effectiveness a priori as well as assess 
their impact. 


Communicate effectively in diverse environments (including multilingual). 


Collaborate With Others 


Demonstrate ability to work effectively and respectfully with diverse teams. 

Exercise flexibility and willingness to be helpful in making necessary compromises to accomplish a common 
goal. 

Assume shared responsibility for collaborative work, and value the individual contributions made by each team 
member. 
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Fourth-grade students can use digital and network tools to find information and identify sources that may be biased 
in some way. Eighth-grade students are able to use digital resources to find information and also to recognize when 
information may be distorted, exaggerated, or otherwise misrepresented. Twelfth-grade students can use advanced 


search methods and select the best digital tools and resources for various purposes, can evaluate information for time- 


B. Information Research 


liness and accuracy, and can check the credibility of sources. 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

1.4.4: Digital and network tools 
and media resources are helpful for 
answering questions, but they can 


sometimes be biased or wrong. 


Students know that: 

1.8.4: Increases in the quantity of 
information available through elec- 
tronic means and the ease by which 
knowledge can be published have 
heightened the need to check sources 
for possible distortion, exaggeration, 


or misrepresentation. 


Students know that: 

1.12.4: Advanced search techniques 
can be used with digital and network 
tools and media resources to locate 
information and to check the credi- 


bility and expertise of sources. 


Students are able to: 
1.4.5: Use digital and network tools 
and media resources to collect, 


organize, and display data in order to 


answer questions and solve problems. 


Students are able to: 

1.8.5: Select and use appropriate 
digital and network tools and media 
resources to collect, organize, ana- 


lyze, and display supporting data to 


answer questions and test hypotheses. 


Students are able to: 

1.12.5: Select digital and network 
tools and media resources to gather 
information and data on a practical 
task, and justify choices based on the 
tools’ efficiency and effectiveness for 


a given purpose. 


1.4.6: Search media and digital 
sources on a community issue and 


identify sources that may be biased. 


1.8.6: Search media and digital 
resources on a community or world 
issue and identify specific examples 
of distortion, exaggeration, or mis- 


representation of information. 


1.12.6: Search media and digital 
resources on a community or world 
issue and evaluate the timeliness and 
accuracy of the information as well as 
the credibility of the source. 


NETS°S Category 3: Research and Information Fluency 


Students apply digital tools to gather, evaluate, and use information. Students: 


a. Plan strategies to guide inquiry. 


b. Locate, organize, analyze, evaluate, synthesize, and ethically use information from a variety of sources and 


media. 


c. Evaluate and select information sources and digital tools based on the appropriateness to specific tasks. 


Process data and report results. 
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Framework for 21st Century Learning: Communication and Collaboration 


Information Literacy 
Access and Evaluate Information 
e Access information efficiently (time) and effectively (sources). 


e Evaluate information critically and competently. 


Use and Manage Information 
e Use information accurately and creatively for the issue or problem at hand. 
e Manage the flow of information from a wide variety of sources. 
e Apply a fundamental understanding of the ethical and legal issues surrounding the access and use of 


information. 
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C. Investigation of Academic and Practical Problems 


Fourth-grade students are able to use digital tools to investigate local issues, test hypotheses, and build models. 
Eighth-grade students are able to use digital tools to investigate alternative solutions to global issues, test moderately 
complicated hypotheses, build models, and conduct simulations. Twelfth-grade students can conduct more sophis- 
ticated investigations and simulations as well as recognize their limitations. For all levels the focus is on types of 


hardware and software rather than on use of particular hardware or software products. 


Grade 4 


Grade 8 


Grade 12 


Students are able to: 
1.4.7: Use digital tools and resources 
to identify and investigate a local 


issue and generate possible solutions. 


Students are able to: 

1.8.7: Use digital tools to identify a 
global issue and investigate possible 
solutions. Select and present the 


most promising sustainable solution. 


Students are able to: 

1.12.7: Use digital tools and 
resources to identify a complicated 
global issue and develop a system- 
atic plan of investigation. Present 
findings in terms of pros and cons of 
two or more innovative sustainable 


solutions. 


1.4.8: Use digital tools to test simple 


hypotheses in various subject areas. 


1.8.8: Use digital tools to gather and 
display data in order to test hypothe- 
ses of moderate complexity in various 


subject areas. Draw and report con- 


clusions consistent with observations. 


1.12.8: Use digital tools to collect, 
analyze, and display data in order 

to design and conduct complicated 
investigations in various subject 
areas. Explain rationale for the design 
and justify conclusions based on 


observed patterns in the data. 


1.4.9: Use digital models to describe 
how parts of a whole interact with 


each other in a model of a system. 


1.8.9: Use a digital model of a system 
to conduct a simulation. Explain 
how changes in the model result in 


different outcomes. 


1.12.9: Having conducted a sim- 
ulation of a system using a digital 
model, draw conclusions about the 
system, or propose possible solutions 
to a problem or ways to reach a goal 
based on outcomes of the simulation. 
Critique the conclusions based on 
the adequacy of the model. 


NETSSS Category 4: Critical Thinking and Decision-Making 


Students use critical-thinking skills to plan and conduct research, manage projects, solve problems, and make 


informed decisions using appropriate digital tools and resources. Students: 


a. Identify and define authentic problems and significant questions for investigation. 
b. Plan and manage activities to develop a solution or complete a project. 

c. Collect and analyze data to identify solutions and/or make informed decisions. 

d 


Use multiple processes and diverse perspectives to explore alternative solutions. 
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Critical Thinking and Problem-Solving 
Reason Effectively 


e Use various types of reasoning (inductive, deductive, etc.) as appropriate to the situation. 


Use Systems Thinking 


e Analyze how parts of a whole interact with each other to produce overall outcomes in complex systems. 


Make Judgments and Decisions 
e Effectively analyze and evaluate evidence, arguments, claims, and beliefs. 
e Analyze and evaluate major alternative points of view. 
e  Synthesize and make connections between information and arguments. 
e Interpret information and draw conclusions based on the best analysis. 


e Reflect critically on learning experiences and processes. 


Solve Problems 
e Solve different kinds of nonfamiliar problems in both conventional and innovative ways. 


e Identify and ask significant questions that clarify various points of view and lead to better solutions. 


Framework for 21st Century Learning: Communication and Collaboration 
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D. Acknowledgment of Ideas and Information 


Fourth-grade students exhibit digital citizenship by understanding that it is permissible to use others’ ideas as long as 


appropriate credit is given but that copyrighted materials cannot be shared freely. Eighth-grade students should be 


aware of and comply with laws and ethical guidelines for incorporating ideas, text, and images into their own work. 


‘Twelfth-grade students should understand the reasons for protecting intellectual property and demonstrate responsi- 


ble and ethical behaviors when using ideas, quotes, and images from others. 


Grade 4 


Grade 8 


Grade 12 


Students know that: 

1.4.10: It is allowable to use other 
people’s ideas in one’s own work pro- 
vided that proper credit is given to 
the original source, whether informa- 
tion is shared in person or through 


ICT media. 


Students know that: 

1.8.10: Style guides provide detailed 
examples for how to give appropriate 
credit to others when incorporating 
their ideas, text, or images in one’s 


own work. 


Students know that: 

1.12.10: Legal requirements govern- 
ing the use of copyrighted infor- 
mation and ethical guidelines for 
appropriate citations are intended to 


protect intellectual property. 


Students are able to: 

1.4.11: Identify or provide examples 
demonstrating respect for copy- 
righted material, such as resisting the 
request from a friend to copy a song 
from a CD or placing copyrighted 


material online. 


Students are able to: 

1.8.11: Identify or provide examples 
of fair use practices that apply appro- 
priate citation of sources when using 
information from books or digital 


resources. 


Students are able to: 

1.12.11: Identify or provide exam- 
ples of responsible and ethical behay- 
ior that follow the letter and spirit 
of current laws concerning personal 
and commercial uses of copyrighted 
material as well as accepted ethi- 

cal practices when using verbatim 
quotes, images, or ideas generated by 


others. 


NETSSS Category 5: Digital Citizenship 


Students understand human, cultural, and societal issues related to technology and practice legal and ethical behavior. 


Students: 


a. Advocate and practice safe, legal, and responsible use of information and technology. 


b. Exhibit a positive attitude toward using technology that supports collaboration, learning, and productivity. 


Demonstrate personal responsibility for lifelong learning. 


d. Exhibit leadership for digital citizenship. 
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Framework for 21st Century Learning: Communication and Collaboration 


Life and Career Skills: Leadership and Responsibility 
Guide and Lead Others 


e Use interpersonal and problem-solving skills to influence and guide others toward a goal. 
e Leverage strengths of others to accomplish a common goal. 
e Inspire others to reach their very best via example and selflessness. 


e Demonstrate integrity and ethical behavior in using influence and power. 


Be Responsible to Others 


e Act responsibly with the interests of the larger community in mind. 
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E. Selection and Use of Digital Tools 


Fourth-grade students know that different digital tools have different purposes and are able to use a number of 
different tools. Eighth-grade students can categorize digital tools by function and can select appropriate tools and 
demonstrate effective use of the tools for different purposes. Twelfth-grade students are competent in the use of a 
broad variety of digital tools and can justify why certain tools are chosen over others that might accomplish the same 


task, by referencing specific features. 


Grade 4 Grade 8 Grade 12 
Students know that: Students know that: Students know that: 
1.4.12: Different digital tools have 1.8.12: Certain digital tools are 1.12.12: A variety of digital tools 
different purposes. appropriate for gathering, orga- exist for a given purpose. The tools 


nizing, analyzing, and presenting 
information, while other kinds of 
tools are appropriate for creating text, 
visualizations, and models and for 


communicating with others. 


differ in features, capacities, oper- 
ating modes, and style. Knowledge 
about many different ICT tools is 
helpful in selecting the best tool for a 
given task. 


Students are able to: 

1.4.13: Use digital tools (appropriate 
for fourth-grade students) effec- 
tively for different purposes, such as 
searching, organizing, and presenting 


information. 


Students are able to: 

1.8.13: Use appropriate digital tools 
to accomplish a variety of tasks, 
including gathering, analyzing, and 
presenting information as well as cre- 
ating text, visualizations, and models 


and communicating with others. 


Students are able to: 

1.12.13: Demonstrate the capability 
to use a variety of digital tools to 
accomplish a task or develop a solu- 
tion for a practical problem. Justify 
the choice of tools, explain why other 
tools were not used based on specific 
features of the tools, and summarize 


the results. 


NETSeS Category 6: Technology Operations and Concepts 


Students demonstrate a sound understanding of technology concepts, systems, and operations. Students: 


a. Understand and use technology systems. 


b. Select and use applications effectively and productively. 


c. Troubleshoot systems and applications. 


d. ‘Transfer current knowledge to learning of new technologies. 
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Framework for 21st Century Learning: Communication and Collaboration 


Apply Technology Effectively 


e Use technology as a tool to research, organize, evaluate, and communicate information. 
e Use digital technologies (computers, PDAs, media players, GPS, etc.), communication/networking tools, 


and social networks appropriately to access, manage, integrate, evaluate, and create information to success- 


fully function in a knowledge economy. 
e Apply a fundamental understanding of the ethical/legal issues surrounding the access and use of informa- 


tion technologies. 
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APPENDIX F: ALIGNMENT TABLE — COMPARING DESIGN AND SYSTEMS 
SUBAREAS TO THE ITEEA STANDARDS FOR TECHNOLOGICAL LITERACY 


2014 - 2018 NAEP 


Standards for Technological Literacy 


A. Nature of Technology 


Fourth graders should know that technology involves 
tools, materials, and creative thinking used to meet 
human needs and wants. Eighth graders should know 
that technology advances through invention and 
innovation and requires a variety of resources. Twelfth 
graders should know how technology coevolves with 
science and other fields to allow people to accomplish 


challenging tasks. 


Standard 1: Students will develop an understanding of 
the characteristics and scope of technology. 


Standard 2: Students will develop an understanding of 
the core concepts of technology. 


Standard 3: Students will develop an understanding of 
the relationships among technologies and the connec- 
tions between technology and other fields of study. 


B. Engineering Design 


Fourth graders should start to answer the question “How 
are technologies created?” by learning to deal with simple 
yet systematic design challenges. Eighth graders should be 
able to use a more elaborate engineering design process, 
including problem definition, the use of prototypes, test- 
ing and iteration, and trade-offs. Twelfth graders should 
have a deep understanding and a broad array of design 


skills, including optimization. 


Standard 8: Students will develop an understanding of 
the attributes of design. 


Standard 9: Students will develop an understanding of 


engineering design. 


Standard 11: Students will develop abilities to apply the 


design process. 
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2014 - 2018 NAEP 
C. Systems Thinking 


Fourth graders should be able to identify systems, sub- 
systems, components, and boundaries in their everyday 
world and to construct simple systems designed to 
accomplish particular goals. Eighth graders should be 
able to describe goals, inputs, outputs, and processes of 
systems, to use reverse engineering and life cycles to ana- 
lyze systems in terms of feedback and the flow of energy, 
and to modify and construct moderately complicated 
systems. Twelfth graders should understand that systems 
are embedded in larger systems, to recognize factors that 
stabilize systems, to use systems for forecasting, and to 


redesign complicated systems to improve reliability. 


Standards for Technological Literacy 


Standard 1: Students will develop an understanding of 
the characteristics and scope of technology. 


Standard 2: Students will develop an understanding of 
the core concepts of technology. 


Standard 13: Students will develop abilities to assess the 
impact of products and systems. 


D. Maintenance and Troubleshooting 


Fourth graders should recognize that tools and machines 
need to be cared for and that devices that fail can be 
fixed or replaced. Eighth graders should know that tools 
and machines must be maintained and be able to use a 
troubleshooting process to diagnose problems in techno- 
logical systems. Twelfth graders should understand the 


importance of maintenance, be able to analyze malfunc- 


tions, and be able to devise ways to reduce future failures. 


Standard 10: Students will develop an understanding 
of the role of troubleshooting, research and develop- 
ment, invention and innovation, and experimentation in 


problem-solving. 


Standard 12: Students will develop abilities to use and 
maintain technological products and systems. 
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APPENDIX G: NAEP TECHNOLOGY AND ENGINEERING LITERACY 
PRELIMINARY ACHIEVEMENT LEVEL DEFINITIONS 


Congress authorized the National Assessment Governing Board to develop appropriate student 
achievement levels on NAEP. The achievement level definitions are statements of what students 
should know and be able to do on NAEP at grades 4, 8, and 12. To fulfill its statutory responsibil- 
ity, the Governing Board developed a policy to guide the development of achievement levels for all 
NAEP subjects. Three levels of achievement were identified to provide the public, educators, and 
policymakers with information on student performance on NAEP. These levels—Basic, Proficient, 
and Advanced—are used as a primary means of reporting NAEP results to describe “how good is 


good enough” at grades 4, 8, and 12. 


Table G-1 displays the Governing Board’s generic policy definitions for Basic, Proficient, and 
Advanced achievement that pertain to all NAEP subjects and grades. 


Table G-1. Generic Achievement Level Policy Definitions 
for the National Assessment of Educational Progress 


Achievement Level Policy Definition 


Advanced This level signifies superior performance. 


This level represents solid academic performance for each grade assessed. Students reaching 


this level have demonstrated competency over challenging subject matter, including subject 


P. jent 
naan matter knowledge, application of such knowledge to real-world situations, and analytical 
skills appropriate to the subject matter. 
Hae This level denotes partial mastery of prerequisite knowledge and skills that are fundamental 
asic 


for proficient work at each grade. 
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During the framework development process the project committees are asked to develop 
preliminary achievement level definitions, based on the generic policy definitions, to guide item 
development. Essentially the purpose of these statements is to provide examples of what students 
performing at the Basic, Proficient, and Advanced achievement levels should know and be able to 
do in terms of the technology and engineering literacy content areas and practices identified in the 
framework. The intended audiences for these preliminary definitions are the NAEP assessment 
development contractor and item writers. The definitions are to be used to ensure that a broad range 
of items is developed at each grade level. Tables G-2, G-3, and G-4 present the preliminary achieve- 
ment level definitions for grades 4, 8, and 12 to clearly illustrate the technology and engineering 


literacy content and practices expected at each grade level. 


The preliminary definitions include illustrative statements selected from the framework’s tech- 
nology and engineering literacy content and practices. The statements are not intended to represent 
the entire set of objectives from the assessment targets, practices, or contexts, nor do the preliminary 
achievement level definitions denote a sense of priority or importance based on the statements 


selected. 


After the assessment is administered, broadly representative panels engage in a standard-setting 
process to determine the achievement level cut scores on the NAEP scale. The cut scores repre- 
sent the minimum score required for performance at each NAEP achievement level. Part of this 
standard-setting process is the development of a set of paragraphs, derived from the preliminary 
achievement level definitions, to be used in reporting the NAEP Technology and Engineering 
Literacy Assessment results to the general public and other audiences. At each grade level there will 
be paragraphs describing what students should know and be able to do at the Basic, Proficient, and 
Advanced levels in terms of the technology and engineering literacy content and practices identified 
in the framework. Cut scores have been finalized for grade 8. (See appendix H.) 


Further information on NAEP achievement levels can be found at www.nagb.org. 
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Table G-2. Grade 4 Preliminary Achievement Level Definitions 


BASIC 


Students know that: 


PROFICIENT 


ADVANCED 


Students know that: Students know that: 


Technology and Society | A new tool, product, or The introduction of a new The introduction of a new tool, 
machine may have benefits and__| tool, product, or machine may product, or machine usually 
costs. change how people live and brings both benefits and costs, 

work. and it may change how people 
live and work. 

Design and Tools help people do their Different tools are better for Different tools are better for 

Systems work, and certain common different purposes. different purposes because they 


tools have particular uses. 


have different features and 


function in different ways. 


Information and 
Communication 


Technology 


Technology and Society 


Design and 
Systems 


Digital and network tools can 


be used to answer questions. 


Students are able to: 


Name two different technolo- 


gies in their own lives. 


Build a model from a kit with 
instructions. 


Digital and network tools and 
media resources are helpful for 
answering questions, but they 
can sometimes be biased or 


wrong. 


Digital and network tools and 
media resources are helpful 

for answering questions, and 
information-gathering can help 
to determine whether they are 


biased or wrong. 


Students are able to: Students are able to: 


Compare the impact of two 
different technologies on their 


own lives. 


Build and test a model to see if 


it works as intended. 


Compare the ways that their 
lives would change if one or 


the other technology were to 


disappear. 


Build and test a model to see if 
it works as intended, and use 
test data to determine how to 


improve the model. 


Information and 
Communication 


Technology 


Use digital and network tools 
and media resources to collect, 


organize, and display data. 


Use digital and network tools 
and media resources to collect, 
organize, and display data in 
order to answer questions and 


solve problems. 


Select and use digital and net- 
work tools and media resources 
to collect, organize, and display 
data in order to answer ques- 


tions and solve problems. 
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Table G-3. Grade 8 Preliminary Achievement Level Definitions 


Technology and Society 


BASIC 


Students know that: 


Technology interacts with soci- 
ety, sometimes bringing about 


changes in a society's economy. 


PROFICIENT 


ADVANCED 


Students know that: Students know that: 


Technology interacts with soci- 
ety, sometimes bringing about 
changes ina society's economy 
and culture, and may lead to 


new needs and wants. 


Technology interacts with soci- 
ety, sometimes bringing about 
changes in a society's economy, 
politics, and culture, and often 
leading to the creation of new 


needs and wants. 


Design and 
Systems 


One tool is better than another 
for a given task as a result of 


prior improvements. 


Tools have been improved over 
time to further the reach of 
hands, voices, memory, and the 


five human senses. 


Tools have been improved over 
time to further the reach of 
hands, voices, memory, and the 
five human senses, and to do so 
more efficiently, accurately, or 


safely. 


Information and 


Some information available 


Increases in the ease by which 


There are various effective 


Communication through electronic means is knowledge can be published ways to check information 
Technology exaggerated or wrong. have heightened the need to available through electronic 
check sources for possible means to identify possible 
distortion, exaggeration, or distortion, exaggeration, or 
misrepresentation. misrepresentation. 
Students are able to: Students are able to: Students are able to: 
Technology and Society | Identify the impacts of a given Identify the impacts of a given Compare the impacts of a given 
technology in a society. technology in a society, and technology on two different 
predict how it might affect a societies, noting factors that 
different society. may make a technology appro- 
priate and sustainable in one 
society but not in another. 
Design and Systems Design and build a sim- Design and build a simple Design and build a simple 


ple model that meets a 


requirement. 


model that meets a require- 
ment, fix it until it works 
(iteration), test it, and gather 
and display data that describe 
its properties using graphs and 
tables. 


model that meets a require- 
ment, fix it until it works 
(iteration), test it, and gather 
and display data that describe 
its properties using graphs and 
tables. Interpret the results to 
suggest improvements and new 


designs. 


Information and 
Communication 


Technology 


Select and use appropriate 
digital and network tools and 
media resources to collect, 


organize, and display data. 


Select and use appropriate 
digital and network tools and 
media resources to collect, 
organize, analyze, and display 
supporting data to answer 
simple questions and test basic 


hypotheses. 


Select and use appropriate 
digital and network tools and 
media resources to collect, 
organize, analyze, and display 
supporting data to answer 
complicated questions and test 
hypotheses, and explain how to 
go about this to others working 


on the same project. 
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Table G-4. Grade 12 Preliminary Achievement Level Definitions 


BASIC PROFICIENT ADVANCED 
Students know that: Students know that: Students know that: 
Technology and Society Changes caused by the introduc- | Changes caused by the Changes caused by the 
tion and use of a new technol- introduction and use of a new introduction and use of a new 
ogy can be gradual or rapid as technology can range from technology can range from 
well as big or small. gradual to rapid and from subtle | gradual to rapid and from subtle 
to obvious, and can change over | to obvious and can change over 
time. time. These changes may vary 
from society to society as a 
result of differences in a society’s 
economy, politics, or culture. 
Design and The development of tools ‘The evolution of tools and The evolution of tools and 
Systems has influenced and advanced materials has played an essential | materials has played an essential 


society. 


role in the development and 
advancement of cities and 


industrial societies. 


role in the advancement of 
civilization, from the establish- 
ment of cities and industrial 
societies to today’s global trade 


and commerce networks. 


Information and 


Advanced search techniques can 


Advanced search techniques 


Advanced search techniques can 


(Comununiertion be used with digital and net- can be used with digital and be used with digital and net- 

Technology work tools and media resources} network tools and media work tools and media resources 
to locate information. resources to locate information | to locate different kinds of 

and to check the credibility and | information and to check cred- 
expertise of sources. ibility and expertise of different 
types of sources. 

Technology and Society | Analyze the cultural, social, Analyze cultural, social, eco- Analyze cultural, social, eco- 
economic, and/or political nomic, and/or political changes | nomic, and/or political changes 
changes that may be triggered that may be triggered by the that may be triggered by the 
by the introduction of a specific | transfer of a specific technology | transfer of a specific technology 
technology into a society. from one society to another. from one society to another. 

Include anticipated and unantic- 
ipated effects. 

Design and Systems Construct and test a model to Construct and test several mod- | Construct and test several 


see if it meets the requirements 
of a problem, then suggest 


improvements. 


els to determine which is best in 
meeting the requirements of a 


problem. 


models to see if they meet the 
requirements of a problem. 
Combine features to achieve the 


best solution. 


Information and Com- 
munication Technology 


Select digital and network tools 
and media resources to gather 
information and data ona 


practical task. 


Select digital and network tools 
and media resources to gather 
information and data on a prac- 
tical task, and justify choices 
based on the tools’ efficiency 
and effectiveness for a given 


purpose. 


Select digital and network tools 
and media resources to gather 
information and data on a prac- 
tical task, justify choices based 
on the tools’ efficiency and 
effectiveness for a given purpose, 
and advise others on which tools 
and resources would best meet 


their needs. 
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APPENDIX H: NAEP TECHNOLOGY AND ENGINEERING LITERACY 
FINALIZED ACHIEVEMENT LEVEL DEFINITIONS AT GRADE 8 


In preparation for the first NAEP Technology and Engineering Literacy Assessment in 2014 to 
be administered to eighth graders nationally, the Governing Board convened panels of experts to 
recommend final achievement level definitions at grade 8. A broadly representative panel of excep- 
tional teachers, educators, and professionals were also convened to engage in a standard-setting 
process to determine the cut scores that correspond to these achievement level definitions. Panelists 
were trained and engaged in a series of discussions designed to ensure informed judgments about 


mapping cut scores to the assessment. 


The finalized achievement levels used for reporting the 2014 NAEP Technology and Engineer- 
ing Literacy Assessment at grade 8 are presented below. The cut scores delineating student perfor- 
mance for each achievement level are also listed in parentheses. These achievement levels and cut 


scores will also be used for reporting results of the 2018 assessment. 


To learn more about NAEP Achievement Levels, see Chapter Five: Reporting Results of the 
NAEP Technology and Engineering Literacy Assessment. 


Technology and Engineering Literacy Achievement Level Descriptions 


Basic (116) 


Eighth-grade students performing at the Basic level should be able to use common tools and 
media to achieve specified goals and identify major impacts. They should demonstrate an under- 
standing that humans can develop solutions by creating and using technologies. They should be able 
to identify major positive and negative effects that technology can have on the natural and designed 
world. Students should be able to use systematic engineering design processes to solve a simple 
problem that responsibly addresses a human need or want. Students should distinguish compo- 
nents in selected technological systems and recognize that technologies require maintenance. They 
should select common information and communications technology tools and media for specified 
purposes, tasks, and audiences. Students should be able to find and evaluate sources, organize and 
display data and other information to address simple research tasks, give appropriate acknowledge- 


ment for use of the work of others, and use feedback from team members (assessed virtually). 
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Proficient (158) 


Eighth-grade students performing at the Proficient level should be able to understand the 
interactions among parts within systems, systematically develop solutions, and contribute to teams 
(assessed virtually) using common and specialized tools to achieve goals. They should be able to 
explain how technology and society influence each other by comparing the benefits and limitations 
of the technologies’ impacts. Students should be able to analyze the interactions among components 
in technological systems and consider how the behavior of a single part affects the whole. They 
should be able to diagnose the cause of a simple technological problem. They should be able to use a 
variety of technologies and work with others using systematic engineering design processes in which 
they iteratively plan, analyze, generate, and communicate solutions. Students should be able to select 
and use an appropriate range of tools and media for a variety of purposes, tasks, and audiences. They 
should be able to contribute to the work of team collaborators (assessed virtually) and provide con- 
structive feedback. Students should be able to find, evaluate, organize, and display data and informa- 
tion to answer research questions, solve problems, and achieve goals, appropriately citing use of the 


ideas, words, and images of others. 


Advanced (209) 


Eighth-grade students performing at the Advanced level should be able to draw upon multiple 
tools and media to address complex problems and goals and demonstrate their understanding of 
the potential impacts on society. They should be able to explain the complex relationships between 
technologies and society and the potential implications of technological decisions on society and 
the natural world. Given criteria and constraints, students should be able to use systematic engi- 
neering design processes to plan, design, and use evidence to evaluate and refine multiple possible 
solutions to a need or problem and justify their solutions. Students should be able to explain the 
relationships among components in technological systems, anticipate maintenance issues, iden- 
tify root causes, and repair faults. They should be able to use a variety of common and specialized 
information technologies to achieve goals, and to produce and communicate solutions to complex 
problems. Students should be able to integrate the use of multiple tools and media, evaluate and use 
data and information, communicate with a range of audiences, and accomplish complex tasks. They 
should be able to use and explain the ethical and appropriate methods for citing use of multimedia 
sources and the ideas and work of others. Students should be able to contribute to collaborative 


tasks on a team (assessed virtually) and organize, monitor, and refine team processes. 
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